2018-06-21 23:23:01,660 DEBUG [main] hbase.HBaseTestingUtility(344): Setting hbase.rootdir to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1 2018-06-21 23:23:01,671 INFO [Time-limited test] hbase.HBaseTestingUtility(954): Starting up minicluster with 1 master(s) and 5 regionserver(s) and 5 datanode(s) 2018-06-21 23:23:01,671 INFO [Time-limited test] hbase.HBaseZKTestingUtility(86): Created new mini-cluster data directory: /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0, deleteOnExit=true 2018-06-21 23:23:01,671 INFO [Time-limited test] hbase.HBaseTestingUtility(969): STARTING DFS 2018-06-21 23:23:01,672 INFO [Time-limited test] hbase.HBaseTestingUtility(746): Setting test.cache.data to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cache_data in system properties and HBase conf 2018-06-21 23:23:01,673 INFO [Time-limited test] hbase.HBaseTestingUtility(746): Setting hadoop.tmp.dir to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/hadoop_tmp in system properties and HBase conf 2018-06-21 23:23:01,673 INFO [Time-limited test] hbase.HBaseTestingUtility(746): Setting hadoop.log.dir to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/hadoop_logs in system properties and HBase conf 2018-06-21 23:23:01,673 INFO [Time-limited test] hbase.HBaseTestingUtility(746): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/mapred_local in system properties and HBase conf 2018-06-21 23:23:01,674 INFO [Time-limited test] hbase.HBaseTestingUtility(746): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/mapred_temp in system properties and HBase conf 2018-06-21 23:23:01,674 INFO [Time-limited test] hbase.HBaseTestingUtility(737): read short circuit is OFF 2018-06-21 23:23:01,804 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-06-21 23:23:02,254 DEBUG [Time-limited test] fs.HFileSystem(317): The file system is not a DistributedFileSystem. Skipping on block location reordering Formatting using clusterid: testClusterID 2018-06-21 23:23:03,757 WARN [Time-limited test] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2018-06-21 23:23:03,976 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2018-06-21 23:23:04,050 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2018-06-21 23:23:04,080 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/yetus-m2/hbase-flaky-tests/org/apache/hadoop/hadoop-hdfs/2.7.4/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to /tmp/Jetty_localhost_37204_hdfs____reo4u4/webapp 2018-06-21 23:23:04,263 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37204 2018-06-21 23:23:05,096 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2018-06-21 23:23:05,103 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/yetus-m2/hbase-flaky-tests/org/apache/hadoop/hadoop-hdfs/2.7.4/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to /tmp/Jetty_localhost_46548_datanode____evliji/webapp 2018-06-21 23:23:05,246 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46548 2018-06-21 23:23:05,562 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2018-06-21 23:23:05,569 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/yetus-m2/hbase-flaky-tests/org/apache/hadoop/hadoop-hdfs/2.7.4/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to /tmp/Jetty_localhost_46673_datanode____.w1ivk9/webapp 2018-06-21 23:23:05,738 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46673 2018-06-21 23:23:05,815 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2018-06-21 23:23:05,822 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/yetus-m2/hbase-flaky-tests/org/apache/hadoop/hadoop-hdfs/2.7.4/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to /tmp/Jetty_localhost_39562_datanode____.kfw1se/webapp 2018-06-21 23:23:05,959 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39562 2018-06-21 23:23:06,030 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2018-06-21 23:23:06,069 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/yetus-m2/hbase-flaky-tests/org/apache/hadoop/hadoop-hdfs/2.7.4/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to /tmp/Jetty_localhost_59991_datanode____wsdff8/webapp 2018-06-21 23:23:06,224 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:59991 2018-06-21 23:23:06,538 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2018-06-21 23:23:06,547 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/yetus-m2/hbase-flaky-tests/org/apache/hadoop/hadoop-hdfs/2.7.4/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to /tmp/Jetty_localhost_42137_datanode____x5zwce/webapp 2018-06-21 23:23:06,679 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42137 2018-06-21 23:23:06,851 ERROR [DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:56139] datanode.DirectoryScanner(430): dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000 2018-06-21 23:23:06,886 ERROR [DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data3/, [DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data4/]] heartbeating to localhost/127.0.0.1:56139] datanode.DirectoryScanner(430): dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000 2018-06-21 23:23:06,955 ERROR [DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data5/, [DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data6/]] heartbeating to localhost/127.0.0.1:56139] datanode.DirectoryScanner(430): dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000 2018-06-21 23:23:07,002 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x1aba277a2fd73e: from storage DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26 node DatanodeRegistration(127.0.0.1:54793, datanodeUuid=937d2ef4-9d52-43ce-9fb6-245f9750c658, infoPort=53852, infoSecurePort=0, ipcPort=53914, storageInfo=lv=-56;cid=testClusterID;nsid=1429840019;c=0), blocks: 0, hasStaleStorage: true, processing time: 2 msecs 2018-06-21 23:23:07,003 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x1aba277a2fd02d: from storage DS-85f0e371-3357-43c0-95ea-ca66d237b4a8 node DatanodeRegistration(127.0.0.1:42074, datanodeUuid=6ec31509-2550-4a57-907a-3986dd3cfb9c, infoPort=58324, infoSecurePort=0, ipcPort=56607, storageInfo=lv=-56;cid=testClusterID;nsid=1429840019;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs 2018-06-21 23:23:07,003 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x1aba277a2fd094: from storage DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d node DatanodeRegistration(127.0.0.1:57095, datanodeUuid=c317d2ee-b8dd-497d-a934-c9aeee7ee76e, infoPort=59889, infoSecurePort=0, ipcPort=38734, storageInfo=lv=-56;cid=testClusterID;nsid=1429840019;c=0), blocks: 0, hasStaleStorage: true, processing time: 1 msecs 2018-06-21 23:23:07,004 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x1aba277a2fd73e: from storage DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e node DatanodeRegistration(127.0.0.1:54793, datanodeUuid=937d2ef4-9d52-43ce-9fb6-245f9750c658, infoPort=53852, infoSecurePort=0, ipcPort=53914, storageInfo=lv=-56;cid=testClusterID;nsid=1429840019;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs 2018-06-21 23:23:07,004 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x1aba277a2fd02d: from storage DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7 node DatanodeRegistration(127.0.0.1:42074, datanodeUuid=6ec31509-2550-4a57-907a-3986dd3cfb9c, infoPort=58324, infoSecurePort=0, ipcPort=56607, storageInfo=lv=-56;cid=testClusterID;nsid=1429840019;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs 2018-06-21 23:23:07,004 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x1aba277a2fd094: from storage DS-901750e1-a133-4f10-85af-4c0c301de1f4 node DatanodeRegistration(127.0.0.1:57095, datanodeUuid=c317d2ee-b8dd-497d-a934-c9aeee7ee76e, infoPort=59889, infoSecurePort=0, ipcPort=38734, storageInfo=lv=-56;cid=testClusterID;nsid=1429840019;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs 2018-06-21 23:23:07,254 ERROR [DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data7/, [DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data8/]] heartbeating to localhost/127.0.0.1:56139] datanode.DirectoryScanner(430): dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000 2018-06-21 23:23:07,263 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x1aba278b136c49: from storage DS-189d75fd-2cba-4781-8f7e-7f1fecbda947 node DatanodeRegistration(127.0.0.1:33402, datanodeUuid=61f88414-0be1-4db2-bf5a-0475ff8fcc50, infoPort=44495, infoSecurePort=0, ipcPort=59929, storageInfo=lv=-56;cid=testClusterID;nsid=1429840019;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs 2018-06-21 23:23:07,263 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x1aba278b136c49: from storage DS-d4d0c936-07f4-4962-b708-add7d68719f3 node DatanodeRegistration(127.0.0.1:33402, datanodeUuid=61f88414-0be1-4db2-bf5a-0475ff8fcc50, infoPort=44495, infoSecurePort=0, ipcPort=59929, storageInfo=lv=-56;cid=testClusterID;nsid=1429840019;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs 2018-06-21 23:23:07,409 ERROR [DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data9/, [DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data10/]] heartbeating to localhost/127.0.0.1:56139] datanode.DirectoryScanner(430): dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000 2018-06-21 23:23:07,420 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x1aba27947650a8: from storage DS-615ebfc4-105b-4470-b106-59b6b4ae84d8 node DatanodeRegistration(127.0.0.1:53553, datanodeUuid=24acd91f-11d1-457c-a0e1-2b33943135cf, infoPort=56927, infoSecurePort=0, ipcPort=55567, storageInfo=lv=-56;cid=testClusterID;nsid=1429840019;c=0), blocks: 0, hasStaleStorage: true, processing time: 1 msecs 2018-06-21 23:23:07,421 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x1aba27947650a8: from storage DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4 node DatanodeRegistration(127.0.0.1:53553, datanodeUuid=24acd91f-11d1-457c-a0e1-2b33943135cf, infoPort=56927, infoSecurePort=0, ipcPort=55567, storageInfo=lv=-56;cid=testClusterID;nsid=1429840019;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs 2018-06-21 23:23:07,553 DEBUG [Time-limited test] hbase.HBaseTestingUtility(672): Setting hbase.rootdir to /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1 2018-06-21 23:23:07,622 ERROR [Time-limited test] server.ZooKeeperServer(472): ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes 2018-06-21 23:23:07,645 INFO [Time-limited test] zookeeper.MiniZooKeeperCluster(281): Started MiniZooKeeperCluster and ran successful 'stat' on client port=58989 2018-06-21 23:23:07,661 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-06-21 23:23:07,663 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-06-21 23:23:08,046 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW]]} size 7 2018-06-21 23:23:08,046 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741825_1001 size 7 2018-06-21 23:23:08,047 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741825_1001 size 7 2018-06-21 23:23:08,463 INFO [Time-limited test] util.FSUtils(515): Created version file at hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba with version=8 2018-06-21 23:23:08,463 INFO [Time-limited test] hbase.HBaseTestingUtility(1213): Setting hbase.fs.tmp.dir to hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/hbase-staging 2018-06-21 23:23:08,699 INFO [Time-limited test] metrics.MetricRegistriesLoader(66): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2018-06-21 23:23:08,963 INFO [Time-limited test] client.ConnectionUtils(122): master/asf911:0 server-side Connection retries=45 2018-06-21 23:23:08,986 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=50, handlerCount=5 2018-06-21 23:23:08,987 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated priority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=60, handlerCount=6 2018-06-21 23:23:08,988 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2018-06-21 23:23:09,127 INFO [Time-limited test] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.ClientService, hbase.pb.AdminService 2018-06-21 23:23:09,341 INFO [Time-limited test] io.ByteBufferPool(83): Created with bufferSize=64 KB and maxPoolSize=320 B 2018-06-21 23:23:09,353 DEBUG [Time-limited test] util.ClassSize(229): Using Unsafe to estimate memory layout 2018-06-21 23:23:09,448 INFO [Time-limited test] ipc.NettyRpcServer(109): Bind to /67.195.81.155:55146 2018-06-21 23:23:09,465 INFO [Time-limited test] hfile.CacheConfig(553): Allocating onheap LruBlockCache size=995.60 MB, blockSize=64 KB 2018-06-21 23:23:09,473 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:09,474 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:09,476 DEBUG [Time-limited test] mob.MobFileCache(123): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2018-06-21 23:23:09,478 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-06-21 23:23:09,481 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-06-21 23:23:09,741 INFO [Time-limited test] zookeeper.RecoverableZooKeeper(106): Process identifier=master:55146 connecting to ZooKeeper ensemble=localhost:58989 2018-06-21 23:23:09,813 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:551460x0, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-06-21 23:23:09,815 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(543): master:55146-0x16424a89dd40000 connected 2018-06-21 23:23:09,937 DEBUG [Time-limited test] zookeeper.ZKUtil(357): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-06-21 23:23:09,939 DEBUG [Time-limited test] zookeeper.ZKUtil(357): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-06-21 23:23:09,953 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=5 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=55146 2018-06-21 23:23:09,954 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=6 with threadPrefix=priority.FPBQ.Fifo, numCallQueues=1, port=55146 2018-06-21 23:23:09,955 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=55146 2018-06-21 23:23:09,962 INFO [Time-limited test] master.HMaster(489): hbase.rootdir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba, hbase.cluster.distributed=false 2018-06-21 23:23:10,068 INFO [Time-limited test] client.ConnectionUtils(122): regionserver/asf911:0 server-side Connection retries=45 2018-06-21 23:23:10,069 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=50, handlerCount=5 2018-06-21 23:23:10,069 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated priority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=60, handlerCount=6 2018-06-21 23:23:10,069 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2018-06-21 23:23:10,074 INFO [Time-limited test] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-06-21 23:23:10,074 INFO [Time-limited test] io.ByteBufferPool(83): Created with bufferSize=64 KB and maxPoolSize=320 B 2018-06-21 23:23:10,085 INFO [Time-limited test] ipc.NettyRpcServer(109): Bind to /67.195.81.155:59970 2018-06-21 23:23:10,086 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:10,088 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:10,092 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-06-21 23:23:10,096 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-06-21 23:23:10,101 INFO [Time-limited test] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:59970 connecting to ZooKeeper ensemble=localhost:58989 2018-06-21 23:23:10,118 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:599700x0, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-06-21 23:23:10,119 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(543): regionserver:59970-0x16424a89dd40001 connected 2018-06-21 23:23:10,119 DEBUG [Time-limited test] zookeeper.ZKUtil(357): regionserver:59970-0x16424a89dd40001, quorum=localhost:58989, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-06-21 23:23:10,120 DEBUG [Time-limited test] zookeeper.ZKUtil(357): regionserver:59970-0x16424a89dd40001, quorum=localhost:58989, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-06-21 23:23:10,121 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=5 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=59970 2018-06-21 23:23:10,122 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=6 with threadPrefix=priority.FPBQ.Fifo, numCallQueues=1, port=59970 2018-06-21 23:23:10,123 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=59970 2018-06-21 23:23:10,150 INFO [Time-limited test] client.ConnectionUtils(122): regionserver/asf911:0 server-side Connection retries=45 2018-06-21 23:23:10,151 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=50, handlerCount=5 2018-06-21 23:23:10,151 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated priority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=60, handlerCount=6 2018-06-21 23:23:10,151 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2018-06-21 23:23:10,151 INFO [Time-limited test] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-06-21 23:23:10,152 INFO [Time-limited test] io.ByteBufferPool(83): Created with bufferSize=64 KB and maxPoolSize=320 B 2018-06-21 23:23:10,155 INFO [Time-limited test] ipc.NettyRpcServer(109): Bind to /67.195.81.155:40767 2018-06-21 23:23:10,156 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:10,158 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:10,161 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-06-21 23:23:10,164 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-06-21 23:23:10,167 INFO [Time-limited test] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:40767 connecting to ZooKeeper ensemble=localhost:58989 2018-06-21 23:23:10,193 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:407670x0, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-06-21 23:23:10,193 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(543): regionserver:40767-0x16424a89dd40002 connected 2018-06-21 23:23:10,194 DEBUG [Time-limited test] zookeeper.ZKUtil(357): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-06-21 23:23:10,195 DEBUG [Time-limited test] zookeeper.ZKUtil(357): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-06-21 23:23:10,196 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=5 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40767 2018-06-21 23:23:10,198 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=6 with threadPrefix=priority.FPBQ.Fifo, numCallQueues=1, port=40767 2018-06-21 23:23:10,198 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40767 2018-06-21 23:23:10,230 INFO [Time-limited test] client.ConnectionUtils(122): regionserver/asf911:0 server-side Connection retries=45 2018-06-21 23:23:10,231 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=50, handlerCount=5 2018-06-21 23:23:10,231 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated priority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=60, handlerCount=6 2018-06-21 23:23:10,232 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2018-06-21 23:23:10,232 INFO [Time-limited test] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-06-21 23:23:10,232 INFO [Time-limited test] io.ByteBufferPool(83): Created with bufferSize=64 KB and maxPoolSize=320 B 2018-06-21 23:23:10,235 INFO [Time-limited test] ipc.NettyRpcServer(109): Bind to /67.195.81.155:52507 2018-06-21 23:23:10,236 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:10,237 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:10,239 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-06-21 23:23:10,242 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-06-21 23:23:10,244 INFO [Time-limited test] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:52507 connecting to ZooKeeper ensemble=localhost:58989 2018-06-21 23:23:10,261 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:525070x0, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-06-21 23:23:10,262 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(543): regionserver:52507-0x16424a89dd40003 connected 2018-06-21 23:23:10,262 DEBUG [Time-limited test] zookeeper.ZKUtil(357): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-06-21 23:23:10,263 DEBUG [Time-limited test] zookeeper.ZKUtil(357): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-06-21 23:23:10,265 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=5 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=52507 2018-06-21 23:23:10,266 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=6 with threadPrefix=priority.FPBQ.Fifo, numCallQueues=1, port=52507 2018-06-21 23:23:10,266 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=52507 2018-06-21 23:23:10,299 INFO [Time-limited test] client.ConnectionUtils(122): regionserver/asf911:0 server-side Connection retries=45 2018-06-21 23:23:10,299 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=50, handlerCount=5 2018-06-21 23:23:10,299 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated priority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=60, handlerCount=6 2018-06-21 23:23:10,299 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2018-06-21 23:23:10,300 INFO [Time-limited test] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-06-21 23:23:10,300 INFO [Time-limited test] io.ByteBufferPool(83): Created with bufferSize=64 KB and maxPoolSize=320 B 2018-06-21 23:23:10,303 INFO [Time-limited test] ipc.NettyRpcServer(109): Bind to /67.195.81.155:56613 2018-06-21 23:23:10,304 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:10,305 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:10,307 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-06-21 23:23:10,310 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-06-21 23:23:10,312 INFO [Time-limited test] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:56613 connecting to ZooKeeper ensemble=localhost:58989 2018-06-21 23:23:10,351 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:566130x0, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-06-21 23:23:10,352 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(543): regionserver:56613-0x16424a89dd40004 connected 2018-06-21 23:23:10,352 DEBUG [Time-limited test] zookeeper.ZKUtil(357): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-06-21 23:23:10,353 DEBUG [Time-limited test] zookeeper.ZKUtil(357): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-06-21 23:23:10,355 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=5 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=56613 2018-06-21 23:23:10,357 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=6 with threadPrefix=priority.FPBQ.Fifo, numCallQueues=1, port=56613 2018-06-21 23:23:10,357 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=56613 2018-06-21 23:23:10,384 INFO [Time-limited test] client.ConnectionUtils(122): regionserver/asf911:0 server-side Connection retries=45 2018-06-21 23:23:10,384 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=50, handlerCount=5 2018-06-21 23:23:10,385 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated priority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=60, handlerCount=6 2018-06-21 23:23:10,385 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2018-06-21 23:23:10,385 INFO [Time-limited test] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-06-21 23:23:10,385 INFO [Time-limited test] io.ByteBufferPool(83): Created with bufferSize=64 KB and maxPoolSize=320 B 2018-06-21 23:23:10,388 INFO [Time-limited test] ipc.NettyRpcServer(109): Bind to /67.195.81.155:58235 2018-06-21 23:23:10,389 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:10,390 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:10,392 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-06-21 23:23:10,396 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-06-21 23:23:10,398 INFO [Time-limited test] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:58235 connecting to ZooKeeper ensemble=localhost:58989 2018-06-21 23:23:10,418 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:582350x0, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-06-21 23:23:10,419 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(543): regionserver:58235-0x16424a89dd40005 connected 2018-06-21 23:23:10,419 DEBUG [Time-limited test] zookeeper.ZKUtil(357): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-06-21 23:23:10,420 DEBUG [Time-limited test] zookeeper.ZKUtil(357): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-06-21 23:23:10,422 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=5 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=58235 2018-06-21 23:23:10,425 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=6 with threadPrefix=priority.FPBQ.Fifo, numCallQueues=1, port=58235 2018-06-21 23:23:10,426 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=58235 2018-06-21 23:23:10,431 INFO [Thread-229] master.HMaster(2100): Adding backup master ZNode /hbase/backup-masters/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:10,452 DEBUG [Thread-229] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:10,517 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-06-21 23:23:10,517 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-06-21 23:23:10,517 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-06-21 23:23:10,517 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:59970-0x16424a89dd40001, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-06-21 23:23:10,517 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-06-21 23:23:10,517 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-06-21 23:23:10,530 DEBUG [Thread-229] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-06-21 23:23:10,536 INFO [Thread-229] master.ActiveMasterManager(172): Deleting ZNode for /hbase/backup-masters/asf911.gq1.ygridcore.net,55146,1529623388534 from backup master directory 2018-06-21 23:23:10,537 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-06-21 23:23:10,551 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:10,553 WARN [Thread-229] hbase.ZNodeClearer(63): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-06-21 23:23:10,553 INFO [Thread-229] master.ActiveMasterManager(181): Registered as active master=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:10,558 INFO [Thread-229] regionserver.ChunkCreator(498): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 448, initial count 0 2018-06-21 23:23:10,560 INFO [Thread-229] regionserver.ChunkCreator(498): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 497, initial count 0 2018-06-21 23:23:10,766 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|FINALIZED]]} size 0 2018-06-21 23:23:10,766 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|FINALIZED], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|FINALIZED]]} size 0 2018-06-21 23:23:10,767 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|FINALIZED], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|FINALIZED]]} size 0 2018-06-21 23:23:10,771 DEBUG [Thread-229] util.FSUtils(667): Created cluster ID file at hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/hbase.id with ID: 2ed7179d-84c9-432a-b6b5-556343aa7690 2018-06-21 23:23:10,818 INFO [Thread-229] master.MasterFileSystem(394): BOOTSTRAP: creating hbase:meta region 2018-06-21 23:23:10,824 INFO [Thread-229] regionserver.HRegion(6917): creating HRegion hbase:meta HTD == 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, {NAME => 'info', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'false', BLOCKSIZE => '8192'}, {NAME => 'rep_barrier', VERSIONS => '2147483647', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'table', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba Table name == hbase:meta 2018-06-21 23:23:10,905 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|FINALIZED]]} size 0 2018-06-21 23:23:10,906 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|FINALIZED]]} size 0 2018-06-21 23:23:10,907 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|FINALIZED]]} size 0 2018-06-21 23:23:10,917 DEBUG [Thread-229] regionserver.HRegion(827): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:10,969 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/info 2018-06-21 23:23:10,995 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:11,007 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:11,025 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:11,030 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/rep_barrier 2018-06-21 23:23:11,032 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:11,033 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:11,034 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:11,039 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/table 2018-06-21 23:23:11,040 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:11,041 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:11,043 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:11,044 DEBUG [Thread-229] regionserver.HRegion(923): replaying wal for 1588230740 2018-06-21 23:23:11,055 DEBUG [Thread-229] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740 2018-06-21 23:23:11,056 DEBUG [Thread-229] regionserver.HRegion(931): stopping wal replay for 1588230740 2018-06-21 23:23:11,057 DEBUG [Thread-229] regionserver.HRegion(943): Cleaning up temporary data for 1588230740 2018-06-21 23:23:11,068 DEBUG [Thread-229] regionserver.HRegion(954): Cleaning up detritus for 1588230740 2018-06-21 23:23:11,077 DEBUG [Thread-229] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7M)) instead. 2018-06-21 23:23:11,079 DEBUG [Thread-229] regionserver.HRegion(976): writing seq id for 1588230740 2018-06-21 23:23:11,088 DEBUG [Thread-229] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-06-21 23:23:11,088 INFO [Thread-229] regionserver.HRegion(980): Opened 1588230740; next sequenceid=2 2018-06-21 23:23:11,088 DEBUG [Thread-229] regionserver.HRegion(1520): Closing 1588230740, disabling compactions & flushes 2018-06-21 23:23:11,088 DEBUG [Thread-229] regionserver.HRegion(1560): Updates disabled for region hbase:meta,,1.1588230740 2018-06-21 23:23:11,090 INFO [Thread-229] regionserver.HRegion(1674): Closed hbase:meta,,1.1588230740 2018-06-21 23:23:11,161 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:11,163 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:11,164 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:11,170 DEBUG [Thread-229] util.FSTableDescriptors(683): Wrote into hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2018-06-21 23:23:11,235 INFO [Thread-229] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-06-21 23:23:11,258 INFO [Thread-229] coordination.ZKSplitLogManagerCoordination(494): Found 0 orphan tasks and 0 rescan nodes 2018-06-21 23:23:11,306 INFO [Thread-229] zookeeper.ReadOnlyZKClient(139): Connect 0x0794c832 to localhost:58989 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-06-21 23:23:11,369 DEBUG [Thread-229] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@71bb662, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-06-21 23:23:11,407 INFO [Thread-229] procedure2.ProcedureExecutor(528): Starting 16 core workers (bigger of cpus/4 or 16) with max (burst) worker count=160 2018-06-21 23:23:11,418 WARN [Thread-229] util.CommonFSUtils$StreamCapabilities(830): Your Hadoop installation does not include the StreamCapabilities class from HDFS-11644, so we will skip checking if any FSDataOutputStreams actually support hflush/hsync. If you are running on top of HDFS this probably just means you have an older version and this can be ignored. If you are running on top of an alternate FileSystem implementation you should manually verify that hflush and hsync are implemented; otherwise you risk data loss and hard to diagnose errors when our assumptions are violated. 2018-06-21 23:23:11,420 DEBUG [Thread-229] util.CommonFSUtils$StreamCapabilities(837): The first request to check for StreamCapabilities came from this stacktrace. java.lang.ClassNotFoundException: org.apache.hadoop.fs.StreamCapabilities at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.util.CommonFSUtils$StreamCapabilities.(CommonFSUtils.java:826) at org.apache.hadoop.hbase.util.CommonFSUtils.hasCapability(CommonFSUtils.java:864) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1042) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:382) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.init(ProcedureExecutor.java:545) at org.apache.hadoop.hbase.master.HMaster.createProcedureExecutor(HMaster.java:1335) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:871) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2120) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:566) at java.lang.Thread.run(Thread.java:748) 2018-06-21 23:23:11,425 INFO [Thread-229] wal.WALProcedureStore(1077): Rolled new Procedure Store WAL, id=1 2018-06-21 23:23:11,428 INFO [Thread-229] procedure2.ProcedureExecutor(547): Recovered WALProcedureStore lease in 18msec 2018-06-21 23:23:11,430 INFO [Thread-229] procedure2.ProcedureExecutor(561): Loaded WALProcedureStore in 1msec 2018-06-21 23:23:11,430 INFO [Thread-229] procedure2.RemoteProcedureDispatcher(97): Instantiated, coreThreads=128 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2018-06-21 23:23:11,466 DEBUG [Thread-229] zookeeper.ZKUtil(614): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-06-21 23:23:11,514 INFO [Thread-229] balancer.BaseLoadBalancer(1039): slop=0.001, tablesOnMaster=true, systemTablesOnMaster=false 2018-06-21 23:23:11,522 INFO [Thread-229] balancer.StochasticLoadBalancer(214): Loaded config; maxSteps=1000000, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, etc. 2018-06-21 23:23:11,546 DEBUG [Thread-229] zookeeper.ZKUtil(357): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2018-06-21 23:23:11,547 DEBUG [Thread-229] zookeeper.ZKUtil(357): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2018-06-21 23:23:11,569 DEBUG [Thread-229] zookeeper.ZKUtil(357): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2018-06-21 23:23:11,570 DEBUG [Thread-229] zookeeper.ZKUtil(357): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2018-06-21 23:23:11,600 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-06-21 23:23:11,600 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:59970-0x16424a89dd40001, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-06-21 23:23:11,602 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-06-21 23:23:11,602 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-06-21 23:23:11,602 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-06-21 23:23:11,600 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-06-21 23:23:11,603 INFO [Thread-229] master.HMaster(780): Active/primary master=asf911.gq1.ygridcore.net,55146,1529623388534, sessionid=0x16424a89dd40000, setting cluster-up flag (Was=false) 2018-06-21 23:23:11,734 DEBUG [Thread-229] procedure.ZKProcedureUtil(272): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2018-06-21 23:23:11,738 DEBUG [Thread-229] procedure.ZKProcedureCoordinator(250): Starting controller for procedure member=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:11,842 DEBUG [Thread-229] procedure.ZKProcedureUtil(272): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2018-06-21 23:23:11,845 DEBUG [Thread-229] procedure.ZKProcedureCoordinator(250): Starting controller for procedure member=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:11,850 INFO [Thread-229] master.ServerManager(1104): No .lastflushedseqids found athdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.lastflushedseqids will record last flushed sequence id for regions by regionserver report all over again 2018-06-21 23:23:11,909 INFO [Thread-229] coprocessor.CoprocessorHost(160): System coprocessor org.apache.hadoop.hbase.client.TestMultiParallel$MyMasterObserver loaded, priority=536870911. 2018-06-21 23:23:11,936 INFO [RS:0;asf911:59970] regionserver.HRegionServer(874): ClusterId : 2ed7179d-84c9-432a-b6b5-556343aa7690 2018-06-21 23:23:11,936 INFO [RS:2;asf911:52507] regionserver.HRegionServer(874): ClusterId : 2ed7179d-84c9-432a-b6b5-556343aa7690 2018-06-21 23:23:11,936 INFO [RS:1;asf911:40767] regionserver.HRegionServer(874): ClusterId : 2ed7179d-84c9-432a-b6b5-556343aa7690 2018-06-21 23:23:11,937 INFO [RS:4;asf911:58235] regionserver.HRegionServer(874): ClusterId : 2ed7179d-84c9-432a-b6b5-556343aa7690 2018-06-21 23:23:11,937 INFO [RS:3;asf911:56613] regionserver.HRegionServer(874): ClusterId : 2ed7179d-84c9-432a-b6b5-556343aa7690 2018-06-21 23:23:11,948 DEBUG [RS:1;asf911:40767] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initializing 2018-06-21 23:23:11,949 DEBUG [RS:0;asf911:59970] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initializing 2018-06-21 23:23:11,949 DEBUG [RS:2;asf911:52507] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initializing 2018-06-21 23:23:11,948 DEBUG [RS:4;asf911:58235] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initializing 2018-06-21 23:23:11,948 DEBUG [RS:3;asf911:56613] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initializing 2018-06-21 23:23:12,012 DEBUG [RS:4;asf911:58235] procedure.RegionServerProcedureManagerHost(47): Procedure flush-table-proc initialized 2018-06-21 23:23:12,012 DEBUG [RS:0;asf911:59970] procedure.RegionServerProcedureManagerHost(47): Procedure flush-table-proc initialized 2018-06-21 23:23:12,012 DEBUG [RS:4;asf911:58235] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initializing 2018-06-21 23:23:12,012 DEBUG [RS:3;asf911:56613] procedure.RegionServerProcedureManagerHost(47): Procedure flush-table-proc initialized 2018-06-21 23:23:12,013 DEBUG [RS:3;asf911:56613] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initializing 2018-06-21 23:23:12,012 DEBUG [RS:2;asf911:52507] procedure.RegionServerProcedureManagerHost(47): Procedure flush-table-proc initialized 2018-06-21 23:23:12,012 DEBUG [RS:1;asf911:40767] procedure.RegionServerProcedureManagerHost(47): Procedure flush-table-proc initialized 2018-06-21 23:23:12,014 DEBUG [RS:1;asf911:40767] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initializing 2018-06-21 23:23:12,013 DEBUG [RS:2;asf911:52507] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initializing 2018-06-21 23:23:12,012 DEBUG [RS:0;asf911:59970] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initializing 2018-06-21 23:23:12,035 DEBUG [RS:1;asf911:40767] procedure.RegionServerProcedureManagerHost(47): Procedure online-snapshot initialized 2018-06-21 23:23:12,035 DEBUG [RS:3;asf911:56613] procedure.RegionServerProcedureManagerHost(47): Procedure online-snapshot initialized 2018-06-21 23:23:12,035 DEBUG [RS:4;asf911:58235] procedure.RegionServerProcedureManagerHost(47): Procedure online-snapshot initialized 2018-06-21 23:23:12,035 DEBUG [RS:0;asf911:59970] procedure.RegionServerProcedureManagerHost(47): Procedure online-snapshot initialized 2018-06-21 23:23:12,035 DEBUG [RS:2;asf911:52507] procedure.RegionServerProcedureManagerHost(47): Procedure online-snapshot initialized 2018-06-21 23:23:12,040 INFO [RS:1;asf911:40767] zookeeper.ReadOnlyZKClient(139): Connect 0x57f0e7d5 to localhost:58989 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-06-21 23:23:12,041 INFO [RS:0;asf911:59970] zookeeper.ReadOnlyZKClient(139): Connect 0x125d969b to localhost:58989 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-06-21 23:23:12,041 INFO [RS:2;asf911:52507] zookeeper.ReadOnlyZKClient(139): Connect 0x6e5097e4 to localhost:58989 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-06-21 23:23:12,041 INFO [RS:3;asf911:56613] zookeeper.ReadOnlyZKClient(139): Connect 0x3934cc14 to localhost:58989 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-06-21 23:23:12,041 INFO [RS:4;asf911:58235] zookeeper.ReadOnlyZKClient(139): Connect 0x5690b764 to localhost:58989 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-06-21 23:23:12,077 DEBUG [RS:2;asf911:52507] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6097645e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-06-21 23:23:12,079 DEBUG [RS:0;asf911:59970] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1c9a7aa0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-06-21 23:23:12,080 DEBUG [RS:1;asf911:40767] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5d088ca5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-06-21 23:23:12,080 DEBUG [RS:2;asf911:52507] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21aa6e65, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-06-21 23:23:12,080 DEBUG [RS:3;asf911:56613] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@33cca764, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-06-21 23:23:12,080 DEBUG [RS:4;asf911:58235] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c68dc14, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-06-21 23:23:12,080 DEBUG [RS:3;asf911:56613] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@76ce2f79, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-06-21 23:23:12,080 DEBUG [RS:1;asf911:40767] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@180a0960, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-06-21 23:23:12,080 DEBUG [RS:0;asf911:59970] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@18a5fdeb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-06-21 23:23:12,081 DEBUG [RS:4;asf911:58235] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@196a59fd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-06-21 23:23:12,086 DEBUG [RS:0;asf911:59970] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:0;asf911:59970 2018-06-21 23:23:12,086 DEBUG [RS:4;asf911:58235] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:4;asf911:58235 2018-06-21 23:23:12,086 DEBUG [RS:2;asf911:52507] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:2;asf911:52507 2018-06-21 23:23:12,086 DEBUG [RS:3;asf911:56613] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:3;asf911:56613 2018-06-21 23:23:12,086 DEBUG [RS:1;asf911:40767] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:1;asf911:40767 2018-06-21 23:23:12,097 INFO [RS:1;asf911:40767] regionserver.RegionServerCoprocessorHost(67): System coprocessor loading is enabled 2018-06-21 23:23:12,097 INFO [RS:4;asf911:58235] regionserver.RegionServerCoprocessorHost(67): System coprocessor loading is enabled 2018-06-21 23:23:12,100 INFO [RS:4;asf911:58235] regionserver.RegionServerCoprocessorHost(68): Table coprocessor loading is enabled 2018-06-21 23:23:12,097 INFO [RS:0;asf911:59970] regionserver.RegionServerCoprocessorHost(67): System coprocessor loading is enabled 2018-06-21 23:23:12,100 INFO [RS:0;asf911:59970] regionserver.RegionServerCoprocessorHost(68): Table coprocessor loading is enabled 2018-06-21 23:23:12,099 INFO [RS:2;asf911:52507] regionserver.RegionServerCoprocessorHost(67): System coprocessor loading is enabled 2018-06-21 23:23:12,100 INFO [RS:2;asf911:52507] regionserver.RegionServerCoprocessorHost(68): Table coprocessor loading is enabled 2018-06-21 23:23:12,097 INFO [RS:3;asf911:56613] regionserver.RegionServerCoprocessorHost(67): System coprocessor loading is enabled 2018-06-21 23:23:12,101 INFO [RS:3;asf911:56613] regionserver.RegionServerCoprocessorHost(68): Table coprocessor loading is enabled 2018-06-21 23:23:12,100 INFO [RS:1;asf911:40767] regionserver.RegionServerCoprocessorHost(68): Table coprocessor loading is enabled 2018-06-21 23:23:12,104 INFO [RS:3;asf911:56613] regionserver.HRegionServer(2600): reportForDuty to master=asf911.gq1.ygridcore.net,55146,1529623388534 with port=56613, startcode=1529623390297 2018-06-21 23:23:12,104 INFO [RS:2;asf911:52507] regionserver.HRegionServer(2600): reportForDuty to master=asf911.gq1.ygridcore.net,55146,1529623388534 with port=52507, startcode=1529623390228 2018-06-21 23:23:12,104 INFO [RS:0;asf911:59970] regionserver.HRegionServer(2600): reportForDuty to master=asf911.gq1.ygridcore.net,55146,1529623388534 with port=59970, startcode=1529623390066 2018-06-21 23:23:12,104 INFO [RS:1;asf911:40767] regionserver.HRegionServer(2600): reportForDuty to master=asf911.gq1.ygridcore.net,55146,1529623388534 with port=40767, startcode=1529623390148 2018-06-21 23:23:12,104 INFO [RS:4;asf911:58235] regionserver.HRegionServer(2600): reportForDuty to master=asf911.gq1.ygridcore.net,55146,1529623388534 with port=58235, startcode=1529623390382 2018-06-21 23:23:12,202 DEBUG [Thread-229] procedure2.ProcedureExecutor(887): Stored pid=1, state=RUNNABLE:INIT_META_ASSIGN_META; InitMetaProcedure table=hbase:meta 2018-06-21 23:23:12,227 DEBUG [Thread-229] executor.ExecutorService(92): Starting executor service name=MASTER_OPEN_REGION-master/asf911:0, corePoolSize=5, maxPoolSize=5 2018-06-21 23:23:12,228 DEBUG [Thread-229] executor.ExecutorService(92): Starting executor service name=MASTER_CLOSE_REGION-master/asf911:0, corePoolSize=5, maxPoolSize=5 2018-06-21 23:23:12,228 DEBUG [Thread-229] executor.ExecutorService(92): Starting executor service name=MASTER_SERVER_OPERATIONS-master/asf911:0, corePoolSize=5, maxPoolSize=5 2018-06-21 23:23:12,228 DEBUG [Thread-229] executor.ExecutorService(92): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/asf911:0, corePoolSize=5, maxPoolSize=5 2018-06-21 23:23:12,228 DEBUG [Thread-229] executor.ExecutorService(92): Starting executor service name=M_LOG_REPLAY_OPS-master/asf911:0, corePoolSize=10, maxPoolSize=10 2018-06-21 23:23:12,229 DEBUG [Thread-229] executor.ExecutorService(92): Starting executor service name=MASTER_TABLE_OPERATIONS-master/asf911:0, corePoolSize=1, maxPoolSize=1 2018-06-21 23:23:12,234 INFO [Thread-229] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1529623422234 2018-06-21 23:23:12,238 INFO [Thread-229] cleaner.CleanerChore$DirScanPool(87): Cleaner pool size is 4 2018-06-21 23:23:12,239 DEBUG [Thread-229] cleaner.CleanerChore(234): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2018-06-21 23:23:12,240 INFO [Thread-229] zookeeper.RecoverableZooKeeper(106): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:58989 2018-06-21 23:23:12,241 DEBUG [Thread-229] cleaner.CleanerChore(234): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2018-06-21 23:23:12,242 DEBUG [Thread-229] cleaner.CleanerChore(234): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2018-06-21 23:23:12,242 INFO [PEWorker-1] procedure2.ProcedureExecutor(1516): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740}] 2018-06-21 23:23:12,244 INFO [Thread-229] cleaner.LogCleaner(122): Creating OldWALs cleaners with size=2 2018-06-21 23:23:12,247 INFO [RS-EventLoopGroup-1-5] ipc.ServerRpcConnection(528): Auth successful for jenkins.hfs.3 (auth:SIMPLE) 2018-06-21 23:23:12,247 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(528): Auth successful for jenkins.hfs.4 (auth:SIMPLE) 2018-06-21 23:23:12,248 INFO [RS-EventLoopGroup-1-6] ipc.ServerRpcConnection(528): Auth successful for jenkins.hfs.1 (auth:SIMPLE) 2018-06-21 23:23:12,248 INFO [RS-EventLoopGroup-1-4] ipc.ServerRpcConnection(528): Auth successful for jenkins.hfs.0 (auth:SIMPLE) 2018-06-21 23:23:12,247 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(528): Auth successful for jenkins.hfs.2 (auth:SIMPLE) 2018-06-21 23:23:12,249 DEBUG [Thread-229] cleaner.CleanerChore(234): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2018-06-21 23:23:12,252 DEBUG [Thread-229-EventThread] zookeeper.ZKWatcher(478): replicationLogCleaner0x0, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-06-21 23:23:12,253 DEBUG [Thread-229-EventThread] zookeeper.ZKWatcher(543): replicationLogCleaner-0x16424a89dd4000c connected 2018-06-21 23:23:12,255 DEBUG [Thread-229] cleaner.CleanerChore(234): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2018-06-21 23:23:12,256 DEBUG [Thread-229] cleaner.CleanerChore(234): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2018-06-21 23:23:12,261 INFO [RS-EventLoopGroup-1-4] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 48152 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:12,261 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 50071 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:12,261 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 50599 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:12,261 INFO [RS-EventLoopGroup-1-6] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 51964 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:12,262 DEBUG [Thread-229] cleaner.HFileCleaner(207): Starting for large file=Thread[Thread-229-HFileCleaner.large.0-1529623392261,5,FailOnTimeoutGroup] 2018-06-21 23:23:12,261 INFO [RS-EventLoopGroup-1-5] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 36326 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:12,262 DEBUG [Thread-229] cleaner.HFileCleaner(222): Starting for small files=Thread[Thread-229-HFileCleaner.small.0-1529623392262,5,FailOnTimeoutGroup] 2018-06-21 23:23:12,286 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=55146] master.ServerManager(439): Registering regionserver=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:12,287 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=55146] master.ServerManager(439): Registering regionserver=asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:12,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=55146] master.ServerManager(439): Registering regionserver=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:12,288 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=55146] master.ServerManager(439): Registering regionserver=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:12,288 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=55146] master.ServerManager(439): Registering regionserver=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:12,315 INFO [PEWorker-10] procedure.MasterProcedureScheduler(696): pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740 checking lock on 1588230740 2018-06-21 23:23:12,316 DEBUG [RS:2;asf911:52507] regionserver.HRegionServer(1505): Config from master: hbase.rootdir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba 2018-06-21 23:23:12,316 DEBUG [RS:3;asf911:56613] regionserver.HRegionServer(1505): Config from master: hbase.rootdir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba 2018-06-21 23:23:12,317 DEBUG [RS:2;asf911:52507] regionserver.HRegionServer(1505): Config from master: fs.defaultFS=hdfs://localhost:56139 2018-06-21 23:23:12,317 DEBUG [RS:2;asf911:52507] regionserver.HRegionServer(1505): Config from master: hbase.master.info.port=-1 2018-06-21 23:23:12,316 DEBUG [RS:4;asf911:58235] regionserver.HRegionServer(1505): Config from master: hbase.rootdir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba 2018-06-21 23:23:12,316 DEBUG [RS:1;asf911:40767] regionserver.HRegionServer(1505): Config from master: hbase.rootdir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba 2018-06-21 23:23:12,316 DEBUG [RS:0;asf911:59970] regionserver.HRegionServer(1505): Config from master: hbase.rootdir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba 2018-06-21 23:23:12,318 DEBUG [RS:1;asf911:40767] regionserver.HRegionServer(1505): Config from master: fs.defaultFS=hdfs://localhost:56139 2018-06-21 23:23:12,317 DEBUG [RS:4;asf911:58235] regionserver.HRegionServer(1505): Config from master: fs.defaultFS=hdfs://localhost:56139 2018-06-21 23:23:12,317 DEBUG [RS:3;asf911:56613] regionserver.HRegionServer(1505): Config from master: fs.defaultFS=hdfs://localhost:56139 2018-06-21 23:23:12,318 DEBUG [RS:4;asf911:58235] regionserver.HRegionServer(1505): Config from master: hbase.master.info.port=-1 2018-06-21 23:23:12,318 DEBUG [RS:1;asf911:40767] regionserver.HRegionServer(1505): Config from master: hbase.master.info.port=-1 2018-06-21 23:23:12,318 DEBUG [RS:0;asf911:59970] regionserver.HRegionServer(1505): Config from master: fs.defaultFS=hdfs://localhost:56139 2018-06-21 23:23:12,318 DEBUG [RS:3;asf911:56613] regionserver.HRegionServer(1505): Config from master: hbase.master.info.port=-1 2018-06-21 23:23:12,319 DEBUG [RS:0;asf911:59970] regionserver.HRegionServer(1505): Config from master: hbase.master.info.port=-1 2018-06-21 23:23:12,326 INFO [PEWorker-10] assignment.AssignProcedure(218): Starting pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2018-06-21 23:23:12,377 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:12,378 DEBUG [RS:2;asf911:52507] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:12,378 DEBUG [RS:3;asf911:56613] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:12,378 WARN [RS:2;asf911:52507] hbase.ZNodeClearer(63): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-06-21 23:23:12,378 DEBUG [RS:1;asf911:40767] zookeeper.ZKUtil(355): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:12,378 WARN [RS:3;asf911:56613] hbase.ZNodeClearer(63): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-06-21 23:23:12,378 DEBUG [RS:0;asf911:59970] zookeeper.ZKUtil(355): regionserver:59970-0x16424a89dd40001, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:12,381 WARN [RS:0;asf911:59970] hbase.ZNodeClearer(63): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-06-21 23:23:12,381 WARN [RS:1;asf911:40767] hbase.ZNodeClearer(63): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-06-21 23:23:12,381 DEBUG [RS:4;asf911:58235] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:12,382 WARN [RS:4;asf911:58235] hbase.ZNodeClearer(63): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-06-21 23:23:12,383 INFO [RegionServerTracker-0] master.RegionServerTracker(170): RegionServer ephemeral node created, adding [asf911.gq1.ygridcore.net,58235,1529623390382] 2018-06-21 23:23:12,383 INFO [RegionServerTracker-0] master.RegionServerTracker(170): RegionServer ephemeral node created, adding [asf911.gq1.ygridcore.net,40767,1529623390148] 2018-06-21 23:23:12,383 INFO [RegionServerTracker-0] master.RegionServerTracker(170): RegionServer ephemeral node created, adding [asf911.gq1.ygridcore.net,52507,1529623390228] 2018-06-21 23:23:12,383 INFO [RegionServerTracker-0] master.RegionServerTracker(170): RegionServer ephemeral node created, adding [asf911.gq1.ygridcore.net,59970,1529623390066] 2018-06-21 23:23:12,384 INFO [RegionServerTracker-0] master.RegionServerTracker(170): RegionServer ephemeral node created, adding [asf911.gq1.ygridcore.net,56613,1529623390297] 2018-06-21 23:23:12,390 INFO [RS:4;asf911:58235] wal.WALFactory(137): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-06-21 23:23:12,390 INFO [RS:2;asf911:52507] wal.WALFactory(137): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-06-21 23:23:12,390 INFO [RS:0;asf911:59970] wal.WALFactory(137): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-06-21 23:23:12,390 INFO [RS:3;asf911:56613] wal.WALFactory(137): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-06-21 23:23:12,390 INFO [RS:1;asf911:40767] wal.WALFactory(137): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-06-21 23:23:12,403 DEBUG [RS:4;asf911:58235] regionserver.HRegionServer(1811): logDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:12,403 DEBUG [RS:2;asf911:52507] regionserver.HRegionServer(1811): logDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:12,404 DEBUG [RS:0;asf911:59970] regionserver.HRegionServer(1811): logDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:12,403 DEBUG [RS:3;asf911:56613] regionserver.HRegionServer(1811): logDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:12,403 DEBUG [RS:1;asf911:40767] regionserver.HRegionServer(1811): logDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:12,458 DEBUG [RS:4;asf911:58235] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:12,458 DEBUG [RS:0;asf911:59970] zookeeper.ZKUtil(355): regionserver:59970-0x16424a89dd40001, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:12,458 DEBUG [RS:1;asf911:40767] zookeeper.ZKUtil(355): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:12,458 DEBUG [RS:3;asf911:56613] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:12,458 DEBUG [RS:2;asf911:52507] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:12,459 DEBUG [RS:4;asf911:58235] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:12,459 DEBUG [RS:0;asf911:59970] zookeeper.ZKUtil(355): regionserver:59970-0x16424a89dd40001, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:12,459 DEBUG [RS:1;asf911:40767] zookeeper.ZKUtil(355): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:12,459 DEBUG [RS:3;asf911:56613] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:12,459 DEBUG [RS:4;asf911:58235] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:12,460 DEBUG [RS:0;asf911:59970] zookeeper.ZKUtil(355): regionserver:59970-0x16424a89dd40001, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:12,459 DEBUG [RS:2;asf911:52507] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:12,460 DEBUG [RS:1;asf911:40767] zookeeper.ZKUtil(355): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:12,460 DEBUG [RS:4;asf911:58235] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:12,460 DEBUG [RS:0;asf911:59970] zookeeper.ZKUtil(355): regionserver:59970-0x16424a89dd40001, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:12,460 DEBUG [RS:3;asf911:56613] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:12,461 DEBUG [RS:2;asf911:52507] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:12,461 DEBUG [RS:1;asf911:40767] zookeeper.ZKUtil(355): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:12,461 DEBUG [RS:4;asf911:58235] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:12,461 DEBUG [RS:0;asf911:59970] zookeeper.ZKUtil(355): regionserver:59970-0x16424a89dd40001, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:12,461 DEBUG [RS:2;asf911:52507] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:12,461 DEBUG [RS:3;asf911:56613] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:12,462 DEBUG [RS:1;asf911:40767] zookeeper.ZKUtil(355): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:12,462 DEBUG [RS:2;asf911:52507] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:12,462 DEBUG [RS:3;asf911:56613] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:12,470 DEBUG [RS:1;asf911:40767] regionserver.Replication(132): Replication stats-in-log period=300 seconds 2018-06-21 23:23:12,470 DEBUG [RS:0;asf911:59970] regionserver.Replication(132): Replication stats-in-log period=300 seconds 2018-06-21 23:23:12,470 DEBUG [RS:2;asf911:52507] regionserver.Replication(132): Replication stats-in-log period=300 seconds 2018-06-21 23:23:12,470 DEBUG [RS:4;asf911:58235] regionserver.Replication(132): Replication stats-in-log period=300 seconds 2018-06-21 23:23:12,470 DEBUG [RS:3;asf911:56613] regionserver.Replication(132): Replication stats-in-log period=300 seconds 2018-06-21 23:23:12,480 DEBUG [master/asf911:0] assignment.AssignmentManager(1703): Processing assignQueue; systemServersCount=5, allServersCount=5 2018-06-21 23:23:12,483 INFO [RS:4;asf911:58235] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-06-21 23:23:12,484 INFO [RS:1;asf911:40767] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-06-21 23:23:12,483 INFO [RS:2;asf911:52507] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-06-21 23:23:12,483 INFO [RS:0;asf911:59970] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-06-21 23:23:12,483 INFO [RS:3;asf911:56613] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-06-21 23:23:12,490 DEBUG [master/asf911:0] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region hbase:meta,,1.1588230740 2018-06-21 23:23:12,541 INFO [PEWorker-12] assignment.AssignProcedure(246): Early suspend! pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:12,564 INFO [RS:1;asf911:40767] regionserver.MemStoreFlusher(133): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-06-21 23:23:12,564 INFO [RS:0;asf911:59970] regionserver.MemStoreFlusher(133): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-06-21 23:23:12,564 INFO [RS:3;asf911:56613] regionserver.MemStoreFlusher(133): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-06-21 23:23:12,564 INFO [RS:4;asf911:58235] regionserver.MemStoreFlusher(133): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-06-21 23:23:12,567 INFO [RS:2;asf911:52507] regionserver.MemStoreFlusher(133): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-06-21 23:23:12,574 INFO [RS:2;asf911:52507] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/second, lower bound 10.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2018-06-21 23:23:12,574 INFO [RS:1;asf911:40767] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/second, lower bound 10.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2018-06-21 23:23:12,574 INFO [RS:0;asf911:59970] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/second, lower bound 10.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2018-06-21 23:23:12,574 INFO [RS:4;asf911:58235] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/second, lower bound 10.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2018-06-21 23:23:12,574 INFO [RS:3;asf911:56613] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/second, lower bound 10.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2018-06-21 23:23:12,578 INFO [RS:2;asf911:52507] regionserver.HRegionServer$CompactionChecker(1705): CompactionChecker runs every PT1S 2018-06-21 23:23:12,578 INFO [RS:0;asf911:59970] regionserver.HRegionServer$CompactionChecker(1705): CompactionChecker runs every PT1S 2018-06-21 23:23:12,578 INFO [RS:3;asf911:56613] regionserver.HRegionServer$CompactionChecker(1705): CompactionChecker runs every PT1S 2018-06-21 23:23:12,578 INFO [RS:4;asf911:58235] regionserver.HRegionServer$CompactionChecker(1705): CompactionChecker runs every PT1S 2018-06-21 23:23:12,578 INFO [RS:1;asf911:40767] regionserver.HRegionServer$CompactionChecker(1705): CompactionChecker runs every PT1S 2018-06-21 23:23:12,592 DEBUG [RS:3;asf911:56613] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:12,592 DEBUG [RS:1;asf911:40767] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:12,592 DEBUG [RS:2;asf911:52507] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:12,592 DEBUG [RS:4;asf911:58235] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:12,592 DEBUG [RS:0;asf911:59970] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:12,593 DEBUG [RS:4;asf911:58235] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-regionserver/asf911:0, corePoolSize=1, maxPoolSize=1 2018-06-21 23:23:12,593 DEBUG [RS:2;asf911:52507] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-regionserver/asf911:0, corePoolSize=1, maxPoolSize=1 2018-06-21 23:23:12,593 DEBUG [RS:1;asf911:40767] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-regionserver/asf911:0, corePoolSize=1, maxPoolSize=1 2018-06-21 23:23:12,592 DEBUG [RS:3;asf911:56613] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-regionserver/asf911:0, corePoolSize=1, maxPoolSize=1 2018-06-21 23:23:12,594 DEBUG [RS:1;asf911:40767] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:12,593 DEBUG [RS:2;asf911:52507] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:12,593 DEBUG [RS:4;asf911:58235] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:12,593 DEBUG [RS:0;asf911:59970] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-regionserver/asf911:0, corePoolSize=1, maxPoolSize=1 2018-06-21 23:23:12,594 DEBUG [RS:4;asf911:58235] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:12,594 DEBUG [RS:2;asf911:52507] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:12,594 DEBUG [RS:1;asf911:40767] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:12,594 DEBUG [RS:3;asf911:56613] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:12,595 DEBUG [RS:1;asf911:40767] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-regionserver/asf911:0, corePoolSize=1, maxPoolSize=1 2018-06-21 23:23:12,595 DEBUG [RS:2;asf911:52507] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-regionserver/asf911:0, corePoolSize=1, maxPoolSize=1 2018-06-21 23:23:12,595 DEBUG [RS:1;asf911:40767] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/asf911:0, corePoolSize=2, maxPoolSize=2 2018-06-21 23:23:12,595 DEBUG [RS:4;asf911:58235] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-regionserver/asf911:0, corePoolSize=1, maxPoolSize=1 2018-06-21 23:23:12,594 DEBUG [RS:0;asf911:59970] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:12,596 DEBUG [RS:4;asf911:58235] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/asf911:0, corePoolSize=2, maxPoolSize=2 2018-06-21 23:23:12,596 DEBUG [RS:1;asf911:40767] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/asf911:0, corePoolSize=10, maxPoolSize=10 2018-06-21 23:23:12,595 DEBUG [RS:2;asf911:52507] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/asf911:0, corePoolSize=2, maxPoolSize=2 2018-06-21 23:23:12,595 DEBUG [RS:3;asf911:56613] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:12,596 DEBUG [RS:2;asf911:52507] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/asf911:0, corePoolSize=10, maxPoolSize=10 2018-06-21 23:23:12,596 DEBUG [RS:1;asf911:40767] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:12,596 DEBUG [RS:4;asf911:58235] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/asf911:0, corePoolSize=10, maxPoolSize=10 2018-06-21 23:23:12,597 DEBUG [RS:1;asf911:40767] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-regionserver/asf911:0, corePoolSize=2, maxPoolSize=2 2018-06-21 23:23:12,596 DEBUG [RS:0;asf911:59970] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:12,597 DEBUG [RS:4;asf911:58235] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:12,597 DEBUG [RS:2;asf911:52507] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:12,597 DEBUG [RS:3;asf911:56613] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-regionserver/asf911:0, corePoolSize=1, maxPoolSize=1 2018-06-21 23:23:12,598 DEBUG [RS:2;asf911:52507] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-regionserver/asf911:0, corePoolSize=2, maxPoolSize=2 2018-06-21 23:23:12,598 DEBUG [RS:4;asf911:58235] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-regionserver/asf911:0, corePoolSize=2, maxPoolSize=2 2018-06-21 23:23:12,597 DEBUG [RS:0;asf911:59970] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-regionserver/asf911:0, corePoolSize=1, maxPoolSize=1 2018-06-21 23:23:12,598 DEBUG [RS:3;asf911:56613] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/asf911:0, corePoolSize=2, maxPoolSize=2 2018-06-21 23:23:12,598 DEBUG [RS:0;asf911:59970] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/asf911:0, corePoolSize=2, maxPoolSize=2 2018-06-21 23:23:12,599 DEBUG [RS:3;asf911:56613] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/asf911:0, corePoolSize=10, maxPoolSize=10 2018-06-21 23:23:12,599 DEBUG [RS:0;asf911:59970] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/asf911:0, corePoolSize=10, maxPoolSize=10 2018-06-21 23:23:12,599 DEBUG [RS:3;asf911:56613] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:12,599 DEBUG [RS:0;asf911:59970] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:12,599 DEBUG [RS:3;asf911:56613] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-regionserver/asf911:0, corePoolSize=2, maxPoolSize=2 2018-06-21 23:23:12,599 DEBUG [RS:0;asf911:59970] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-regionserver/asf911:0, corePoolSize=2, maxPoolSize=2 2018-06-21 23:23:12,606 DEBUG [M:0;asf911:55146] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initializing 2018-06-21 23:23:12,618 DEBUG [M:0;asf911:55146] procedure.RegionServerProcedureManagerHost(47): Procedure flush-table-proc initialized 2018-06-21 23:23:12,618 DEBUG [M:0;asf911:55146] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initializing 2018-06-21 23:23:12,626 DEBUG [M:0;asf911:55146] procedure.RegionServerProcedureManagerHost(47): Procedure online-snapshot initialized 2018-06-21 23:23:12,629 DEBUG [M:0;asf911:55146] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@639ed3f7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-06-21 23:23:12,629 DEBUG [M:0;asf911:55146] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:M:0;asf911:55146 2018-06-21 23:23:12,630 INFO [M:0;asf911:55146] regionserver.RegionServerCoprocessorHost(67): System coprocessor loading is enabled 2018-06-21 23:23:12,630 INFO [M:0;asf911:55146] regionserver.RegionServerCoprocessorHost(68): Table coprocessor loading is enabled 2018-06-21 23:23:12,630 INFO [M:0;asf911:55146] regionserver.HRegionServer(2600): reportForDuty to master=asf911.gq1.ygridcore.net,55146,1529623388534 with port=55146, startcode=1529623388534 2018-06-21 23:23:12,631 INFO [M:0;asf911:55146] master.ServerManager(439): Registering regionserver=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:12,631 DEBUG [M:0;asf911:55146] regionserver.HRegionServer(1505): Config from master: hbase.rootdir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba 2018-06-21 23:23:12,631 DEBUG [M:0;asf911:55146] regionserver.HRegionServer(1505): Config from master: fs.defaultFS=hdfs://localhost:56139 2018-06-21 23:23:12,631 DEBUG [M:0;asf911:55146] regionserver.HRegionServer(1505): Config from master: hbase.master.info.port=-1 2018-06-21 23:23:12,634 INFO [SplitLogWorker-asf911:40767] regionserver.SplitLogWorker(136): SplitLogWorker asf911.gq1.ygridcore.net,40767,1529623390148 starting 2018-06-21 23:23:12,634 INFO [SplitLogWorker-asf911:52507] regionserver.SplitLogWorker(136): SplitLogWorker asf911.gq1.ygridcore.net,52507,1529623390228 starting 2018-06-21 23:23:12,635 INFO [SplitLogWorker-asf911:59970] regionserver.SplitLogWorker(136): SplitLogWorker asf911.gq1.ygridcore.net,59970,1529623390066 starting 2018-06-21 23:23:12,637 INFO [SplitLogWorker-asf911:58235] regionserver.SplitLogWorker(136): SplitLogWorker asf911.gq1.ygridcore.net,58235,1529623390382 starting 2018-06-21 23:23:12,639 INFO [RS:4;asf911:58235] regionserver.HeapMemoryManager(210): Starting, tuneOn=false 2018-06-21 23:23:12,639 INFO [RS:1;asf911:40767] regionserver.HeapMemoryManager(210): Starting, tuneOn=false 2018-06-21 23:23:12,639 INFO [RS:2;asf911:52507] regionserver.HeapMemoryManager(210): Starting, tuneOn=false 2018-06-21 23:23:12,639 INFO [RS:0;asf911:59970] regionserver.HeapMemoryManager(210): Starting, tuneOn=false 2018-06-21 23:23:12,642 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:12,642 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:12,643 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:12,643 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:12,643 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:12,643 INFO [SplitLogWorker-asf911:56613] regionserver.SplitLogWorker(136): SplitLogWorker asf911.gq1.ygridcore.net,56613,1529623390297 starting 2018-06-21 23:23:12,643 INFO [RS:3;asf911:56613] regionserver.HeapMemoryManager(210): Starting, tuneOn=false 2018-06-21 23:23:12,643 DEBUG [M:0;asf911:55146] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:12,644 WARN [M:0;asf911:55146] hbase.ZNodeClearer(63): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-06-21 23:23:12,644 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:59970-0x16424a89dd40001, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:12,644 INFO [M:0;asf911:55146] wal.WALFactory(137): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-06-21 23:23:12,646 INFO [RegionServerTracker-0] master.RegionServerTracker(170): RegionServer ephemeral node created, adding [asf911.gq1.ygridcore.net,55146,1529623388534] 2018-06-21 23:23:12,646 DEBUG [M:0;asf911:55146] regionserver.HRegionServer(1811): logDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:12,647 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:12,647 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:12,647 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:12,647 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:59970-0x16424a89dd40001, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:12,648 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:12,648 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:12,648 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:12,649 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:12,649 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:59970-0x16424a89dd40001, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:12,649 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:12,650 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:12,650 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:12,651 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:12,651 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:59970-0x16424a89dd40001, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:12,651 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:12,651 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:12,652 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:12,652 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:59970-0x16424a89dd40001, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:12,658 DEBUG [M:0;asf911:55146] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:12,658 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:12,662 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:12,665 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:12,666 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:59970-0x16424a89dd40001, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:12,666 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:12,666 DEBUG [M:0;asf911:55146] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:12,667 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:12,668 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:12,669 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:12,669 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:59970-0x16424a89dd40001, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:12,671 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:12,673 DEBUG [M:0;asf911:55146] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:12,674 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:12,674 DEBUG [M:0;asf911:55146] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:12,675 DEBUG [M:0;asf911:55146] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:12,675 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:12,677 DEBUG [M:0;asf911:55146] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:12,679 DEBUG [M:0;asf911:55146] regionserver.Replication(132): Replication stats-in-log period=300 seconds 2018-06-21 23:23:12,680 INFO [M:0;asf911:55146] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-06-21 23:23:12,696 INFO [M:0;asf911:55146] regionserver.MemStoreFlusher(133): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-06-21 23:23:12,697 INFO [M:0;asf911:55146] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/second, lower bound 10.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2018-06-21 23:23:12,697 INFO [M:0;asf911:55146] regionserver.HRegionServer$CompactionChecker(1705): CompactionChecker runs every PT1S 2018-06-21 23:23:12,702 DEBUG [M:0;asf911:55146] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-master/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:12,703 DEBUG [M:0;asf911:55146] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-master/asf911:0, corePoolSize=1, maxPoolSize=1 2018-06-21 23:23:12,703 DEBUG [M:0;asf911:55146] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-master/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:12,703 DEBUG [M:0;asf911:55146] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-master/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:12,704 DEBUG [M:0;asf911:55146] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-master/asf911:0, corePoolSize=1, maxPoolSize=1 2018-06-21 23:23:12,704 DEBUG [M:0;asf911:55146] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-master/asf911:0, corePoolSize=2, maxPoolSize=2 2018-06-21 23:23:12,705 DEBUG [M:0;asf911:55146] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-master/asf911:0, corePoolSize=10, maxPoolSize=10 2018-06-21 23:23:12,705 DEBUG [M:0;asf911:55146] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-master/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:12,705 DEBUG [M:0;asf911:55146] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-master/asf911:0, corePoolSize=2, maxPoolSize=2 2018-06-21 23:23:12,722 INFO [RS:4;asf911:58235] regionserver.HRegionServer(1546): Serving as asf911.gq1.ygridcore.net,58235,1529623390382, RpcServer on asf911.gq1.ygridcore.net/67.195.81.155:58235, sessionid=0x16424a89dd40005 2018-06-21 23:23:12,722 INFO [RS:3;asf911:56613] regionserver.HRegionServer(1546): Serving as asf911.gq1.ygridcore.net,56613,1529623390297, RpcServer on asf911.gq1.ygridcore.net/67.195.81.155:56613, sessionid=0x16424a89dd40004 2018-06-21 23:23:12,722 INFO [RS:1;asf911:40767] regionserver.HRegionServer(1546): Serving as asf911.gq1.ygridcore.net,40767,1529623390148, RpcServer on asf911.gq1.ygridcore.net/67.195.81.155:40767, sessionid=0x16424a89dd40002 2018-06-21 23:23:12,722 INFO [RS:2;asf911:52507] regionserver.HRegionServer(1546): Serving as asf911.gq1.ygridcore.net,52507,1529623390228, RpcServer on asf911.gq1.ygridcore.net/67.195.81.155:52507, sessionid=0x16424a89dd40003 2018-06-21 23:23:12,722 INFO [RS:0;asf911:59970] regionserver.HRegionServer(1546): Serving as asf911.gq1.ygridcore.net,59970,1529623390066, RpcServer on asf911.gq1.ygridcore.net/67.195.81.155:59970, sessionid=0x16424a89dd40001 2018-06-21 23:23:12,723 DEBUG [RS:2;asf911:52507] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc starting 2018-06-21 23:23:12,723 DEBUG [RS:1;asf911:40767] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc starting 2018-06-21 23:23:12,723 DEBUG [RS:3;asf911:56613] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc starting 2018-06-21 23:23:12,723 DEBUG [RS:1;asf911:40767] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:12,723 DEBUG [RS:2;asf911:52507] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:12,723 DEBUG [RS:2;asf911:52507] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'asf911.gq1.ygridcore.net,52507,1529623390228' 2018-06-21 23:23:12,723 DEBUG [RS:4;asf911:58235] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc starting 2018-06-21 23:23:12,723 DEBUG [RS:4;asf911:58235] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:12,724 DEBUG [RS:4;asf911:58235] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'asf911.gq1.ygridcore.net,58235,1529623390382' 2018-06-21 23:23:12,724 DEBUG [RS:4;asf911:58235] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-06-21 23:23:12,723 DEBUG [RS:0;asf911:59970] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc starting 2018-06-21 23:23:12,723 DEBUG [RS:2;asf911:52507] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-06-21 23:23:12,723 DEBUG [RS:1;asf911:40767] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'asf911.gq1.ygridcore.net,40767,1529623390148' 2018-06-21 23:23:12,723 DEBUG [RS:3;asf911:56613] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:12,724 DEBUG [RS:1;asf911:40767] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-06-21 23:23:12,724 DEBUG [RS:0;asf911:59970] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:12,724 DEBUG [RS:0;asf911:59970] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'asf911.gq1.ygridcore.net,59970,1529623390066' 2018-06-21 23:23:12,725 DEBUG [RS:0;asf911:59970] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-06-21 23:23:12,725 DEBUG [RS:2;asf911:52507] procedure.ZKProcedureMemberRpcs(155): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-06-21 23:23:12,724 DEBUG [RS:3;asf911:56613] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'asf911.gq1.ygridcore.net,56613,1529623390297' 2018-06-21 23:23:12,725 DEBUG [RS:1;asf911:40767] procedure.ZKProcedureMemberRpcs(155): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-06-21 23:23:12,725 DEBUG [RS:4;asf911:58235] procedure.ZKProcedureMemberRpcs(155): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-06-21 23:23:12,725 DEBUG [RS:3;asf911:56613] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-06-21 23:23:12,725 DEBUG [RS:0;asf911:59970] procedure.ZKProcedureMemberRpcs(155): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-06-21 23:23:12,726 DEBUG [RS:1;asf911:40767] procedure.RegionServerProcedureManagerHost(55): Procedure flush-table-proc started 2018-06-21 23:23:12,726 DEBUG [RS:2;asf911:52507] procedure.RegionServerProcedureManagerHost(55): Procedure flush-table-proc started 2018-06-21 23:23:12,726 DEBUG [RS:1;asf911:40767] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot starting 2018-06-21 23:23:12,726 DEBUG [RS:4;asf911:58235] procedure.RegionServerProcedureManagerHost(55): Procedure flush-table-proc started 2018-06-21 23:23:12,726 DEBUG [RS:1;asf911:40767] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:12,726 DEBUG [RS:1;asf911:40767] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'asf911.gq1.ygridcore.net,40767,1529623390148' 2018-06-21 23:23:12,726 DEBUG [RS:1;asf911:40767] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-06-21 23:23:12,726 DEBUG [RS:3;asf911:56613] procedure.ZKProcedureMemberRpcs(155): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-06-21 23:23:12,726 DEBUG [RS:2;asf911:52507] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot starting 2018-06-21 23:23:12,727 DEBUG [RS:2;asf911:52507] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:12,727 DEBUG [RS:2;asf911:52507] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'asf911.gq1.ygridcore.net,52507,1529623390228' 2018-06-21 23:23:12,726 DEBUG [RS:0;asf911:59970] procedure.RegionServerProcedureManagerHost(55): Procedure flush-table-proc started 2018-06-21 23:23:12,727 DEBUG [RS:3;asf911:56613] procedure.RegionServerProcedureManagerHost(55): Procedure flush-table-proc started 2018-06-21 23:23:12,726 DEBUG [RS:4;asf911:58235] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot starting 2018-06-21 23:23:12,727 DEBUG [RS:4;asf911:58235] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:12,727 DEBUG [RS:4;asf911:58235] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'asf911.gq1.ygridcore.net,58235,1529623390382' 2018-06-21 23:23:12,728 DEBUG [RS:4;asf911:58235] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-06-21 23:23:12,727 DEBUG [RS:3;asf911:56613] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot starting 2018-06-21 23:23:12,728 DEBUG [RS:3;asf911:56613] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:12,728 DEBUG [RS:3;asf911:56613] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'asf911.gq1.ygridcore.net,56613,1529623390297' 2018-06-21 23:23:12,728 DEBUG [RS:3;asf911:56613] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-06-21 23:23:12,727 DEBUG [RS:0;asf911:59970] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot starting 2018-06-21 23:23:12,728 DEBUG [RS:0;asf911:59970] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:12,728 DEBUG [RS:0;asf911:59970] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'asf911.gq1.ygridcore.net,59970,1529623390066' 2018-06-21 23:23:12,728 DEBUG [RS:0;asf911:59970] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-06-21 23:23:12,729 DEBUG [RS:3;asf911:56613] procedure.ZKProcedureMemberRpcs(155): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-06-21 23:23:12,727 DEBUG [RS:2;asf911:52507] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-06-21 23:23:12,727 DEBUG [RS:1;asf911:40767] procedure.ZKProcedureMemberRpcs(155): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-06-21 23:23:12,728 DEBUG [RS:4;asf911:58235] procedure.ZKProcedureMemberRpcs(155): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-06-21 23:23:12,729 DEBUG [RS:0;asf911:59970] procedure.ZKProcedureMemberRpcs(155): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-06-21 23:23:12,729 DEBUG [RS:3;asf911:56613] procedure.RegionServerProcedureManagerHost(55): Procedure online-snapshot started 2018-06-21 23:23:12,729 DEBUG [RS:1;asf911:40767] procedure.RegionServerProcedureManagerHost(55): Procedure online-snapshot started 2018-06-21 23:23:12,729 INFO [RS:1;asf911:40767] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-06-21 23:23:12,729 INFO [RS:3;asf911:56613] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-06-21 23:23:12,729 DEBUG [RS:2;asf911:52507] procedure.ZKProcedureMemberRpcs(155): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-06-21 23:23:12,730 DEBUG [RS:0;asf911:59970] procedure.RegionServerProcedureManagerHost(55): Procedure online-snapshot started 2018-06-21 23:23:12,730 DEBUG [RS:4;asf911:58235] procedure.RegionServerProcedureManagerHost(55): Procedure online-snapshot started 2018-06-21 23:23:12,730 INFO [RS:4;asf911:58235] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-06-21 23:23:12,730 INFO [RS:4;asf911:58235] quotas.RegionServerSpaceQuotaManager(84): Quota support disabled, not starting space quota manager. 2018-06-21 23:23:12,730 INFO [RS:3;asf911:56613] quotas.RegionServerSpaceQuotaManager(84): Quota support disabled, not starting space quota manager. 2018-06-21 23:23:12,730 INFO [RS:1;asf911:40767] quotas.RegionServerSpaceQuotaManager(84): Quota support disabled, not starting space quota manager. 2018-06-21 23:23:12,730 DEBUG [RS:2;asf911:52507] procedure.RegionServerProcedureManagerHost(55): Procedure online-snapshot started 2018-06-21 23:23:12,730 INFO [RS:0;asf911:59970] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-06-21 23:23:12,731 INFO [RS:2;asf911:52507] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-06-21 23:23:12,731 INFO [RS:0;asf911:59970] quotas.RegionServerSpaceQuotaManager(84): Quota support disabled, not starting space quota manager. 2018-06-21 23:23:12,731 INFO [RS:2;asf911:52507] quotas.RegionServerSpaceQuotaManager(84): Quota support disabled, not starting space quota manager. 2018-06-21 23:23:12,740 INFO [SplitLogWorker-asf911:55146] regionserver.SplitLogWorker(136): SplitLogWorker asf911.gq1.ygridcore.net,55146,1529623388534 starting 2018-06-21 23:23:12,740 INFO [M:0;asf911:55146] regionserver.HeapMemoryManager(210): Starting, tuneOn=false 2018-06-21 23:23:12,764 INFO [M:0;asf911:55146] regionserver.HRegionServer(1546): Serving as asf911.gq1.ygridcore.net,55146,1529623388534, RpcServer on asf911.gq1.ygridcore.net/67.195.81.155:55146, sessionid=0x16424a89dd40000 2018-06-21 23:23:12,764 DEBUG [M:0;asf911:55146] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc starting 2018-06-21 23:23:12,765 DEBUG [M:0;asf911:55146] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:12,765 DEBUG [M:0;asf911:55146] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'asf911.gq1.ygridcore.net,55146,1529623388534' 2018-06-21 23:23:12,765 DEBUG [M:0;asf911:55146] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-06-21 23:23:12,766 DEBUG [M:0;asf911:55146] procedure.ZKProcedureMemberRpcs(155): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-06-21 23:23:12,766 DEBUG [M:0;asf911:55146] procedure.RegionServerProcedureManagerHost(55): Procedure flush-table-proc started 2018-06-21 23:23:12,766 DEBUG [M:0;asf911:55146] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot starting 2018-06-21 23:23:12,766 DEBUG [M:0;asf911:55146] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:12,766 DEBUG [M:0;asf911:55146] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'asf911.gq1.ygridcore.net,55146,1529623388534' 2018-06-21 23:23:12,767 DEBUG [M:0;asf911:55146] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-06-21 23:23:12,767 DEBUG [M:0;asf911:55146] procedure.ZKProcedureMemberRpcs(155): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-06-21 23:23:12,768 DEBUG [M:0;asf911:55146] procedure.RegionServerProcedureManagerHost(55): Procedure online-snapshot started 2018-06-21 23:23:12,768 INFO [M:0;asf911:55146] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-06-21 23:23:12,768 INFO [M:0;asf911:55146] quotas.RegionServerSpaceQuotaManager(84): Quota support disabled, not starting space quota manager. 2018-06-21 23:23:13,770 INFO [RS:1;asf911:40767] wal.AbstractFSWAL(424): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C40767%2C1529623390148, suffix=, logDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,40767,1529623390148, archiveDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/oldWALs 2018-06-21 23:23:13,770 INFO [RS:2;asf911:52507] wal.AbstractFSWAL(424): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C52507%2C1529623390228, suffix=, logDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,52507,1529623390228, archiveDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/oldWALs 2018-06-21 23:23:13,770 INFO [RS:4;asf911:58235] wal.AbstractFSWAL(424): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C58235%2C1529623390382, suffix=, logDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,58235,1529623390382, archiveDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/oldWALs 2018-06-21 23:23:13,770 INFO [RS:3;asf911:56613] wal.AbstractFSWAL(424): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C56613%2C1529623390297, suffix=, logDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,56613,1529623390297, archiveDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/oldWALs 2018-06-21 23:23:13,770 INFO [RS:0;asf911:59970] wal.AbstractFSWAL(424): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C59970%2C1529623390066, suffix=, logDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,59970,1529623390066, archiveDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/oldWALs 2018-06-21 23:23:13,773 INFO [M:0;asf911:55146] wal.AbstractFSWAL(424): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C55146%2C1529623388534, suffix=, logDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,55146,1529623388534, archiveDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/oldWALs 2018-06-21 23:23:13,812 DEBUG [RS:4;asf911:58235] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(421): org.apache.hadoop.hdfs.protocolPB.PBHelperClient not found (Hadoop is pre-2.8.0?); using class org.apache.hadoop.hdfs.protocolPB.PBHelper instead. 2018-06-21 23:23:13,867 DEBUG [RS:4;asf911:58235] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(494): No DfsClientConf class found, should be hadoop 2.7- java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.client.impl.DfsClientConf at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createChecksumCreater(FanOutOneBlockAsyncDFSOutputHelper.java:492) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:556) at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:51) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:167) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:165) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:110) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:612) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:124) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:763) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:493) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.(AsyncFSWAL.java:251) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:70) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:45) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:138) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:57) at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:254) at org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:2109) at org.apache.hadoop.hbase.regionserver.HRegionServer.buildServerLoad(HRegionServer.java:1325) at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1191) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1007) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:183) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:129) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:164) at java.lang.Thread.run(Thread.java:748) 2018-06-21 23:23:13,868 DEBUG [RS:4;asf911:58235] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(528): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2018-06-21 23:23:13,941 DEBUG [RS-EventLoopGroup-7-7] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(199): No PBHelperClient class found, should be hadoop 2.7- java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.protocolPB.PBHelperClient at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createPBHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:197) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:261) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:661) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:720) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$13.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:715) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:638) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:676) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:552) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) 2018-06-21 23:23:13,947 DEBUG [RS-EventLoopGroup-7-7] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:54793,DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26,DISK] 2018-06-21 23:23:13,947 DEBUG [RS-EventLoopGroup-7-13] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33402,DS-189d75fd-2cba-4781-8f7e-7f1fecbda947,DISK] 2018-06-21 23:23:13,948 DEBUG [RS-EventLoopGroup-7-11] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:57095,DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d,DISK] 2018-06-21 23:23:13,956 DEBUG [RS-EventLoopGroup-7-15] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:53553,DS-615ebfc4-105b-4470-b106-59b6b4ae84d8,DISK] 2018-06-21 23:23:13,956 DEBUG [RS-EventLoopGroup-7-16] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:54793,DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e,DISK] 2018-06-21 23:23:13,960 DEBUG [RS-EventLoopGroup-7-21] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42074,DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7,DISK] 2018-06-21 23:23:13,960 DEBUG [RS-EventLoopGroup-7-14] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:57095,DS-901750e1-a133-4f10-85af-4c0c301de1f4,DISK] 2018-06-21 23:23:13,961 DEBUG [RS-EventLoopGroup-7-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33402,DS-d4d0c936-07f4-4962-b708-add7d68719f3,DISK] 2018-06-21 23:23:13,961 DEBUG [RS-EventLoopGroup-7-9] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:57095,DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d,DISK] 2018-06-21 23:23:13,965 DEBUG [RS-EventLoopGroup-7-19] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33402,DS-189d75fd-2cba-4781-8f7e-7f1fecbda947,DISK] 2018-06-21 23:23:13,965 DEBUG [RS-EventLoopGroup-7-20] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:57095,DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d,DISK] 2018-06-21 23:23:13,966 DEBUG [RS-EventLoopGroup-7-17] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:53553,DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4,DISK] 2018-06-21 23:23:13,966 DEBUG [RS-EventLoopGroup-7-10] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:53553,DS-615ebfc4-105b-4470-b106-59b6b4ae84d8,DISK] 2018-06-21 23:23:13,966 DEBUG [RS-EventLoopGroup-7-12] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33402,DS-d4d0c936-07f4-4962-b708-add7d68719f3,DISK] 2018-06-21 23:23:13,966 DEBUG [RS-EventLoopGroup-7-18] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33402,DS-d4d0c936-07f4-4962-b708-add7d68719f3,DISK] 2018-06-21 23:23:13,966 DEBUG [RS-EventLoopGroup-7-6] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:57095,DS-901750e1-a133-4f10-85af-4c0c301de1f4,DISK] 2018-06-21 23:23:13,966 DEBUG [RS-EventLoopGroup-7-8] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42074,DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7,DISK] 2018-06-21 23:23:13,981 DEBUG [RS-EventLoopGroup-7-8] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:54793,DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e,DISK] 2018-06-21 23:23:14,064 INFO [RS:0;asf911:59970] wal.AbstractFSWAL(679): New WAL /user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,59970,1529623390066/asf911.gq1.ygridcore.net%2C59970%2C1529623390066.1529623393801 2018-06-21 23:23:14,064 INFO [RS:4;asf911:58235] wal.AbstractFSWAL(679): New WAL /user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,58235,1529623390382/asf911.gq1.ygridcore.net%2C58235%2C1529623390382.1529623393801 2018-06-21 23:23:14,064 INFO [RS:3;asf911:56613] wal.AbstractFSWAL(679): New WAL /user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,56613,1529623390297/asf911.gq1.ygridcore.net%2C56613%2C1529623390297.1529623393801 2018-06-21 23:23:14,064 INFO [RS:2;asf911:52507] wal.AbstractFSWAL(679): New WAL /user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,52507,1529623390228/asf911.gq1.ygridcore.net%2C52507%2C1529623390228.1529623393801 2018-06-21 23:23:14,064 INFO [RS:1;asf911:40767] wal.AbstractFSWAL(679): New WAL /user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,40767,1529623390148/asf911.gq1.ygridcore.net%2C40767%2C1529623390148.1529623393801 2018-06-21 23:23:14,064 INFO [M:0;asf911:55146] wal.AbstractFSWAL(679): New WAL /user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,55146,1529623388534/asf911.gq1.ygridcore.net%2C55146%2C1529623388534.1529623393801 2018-06-21 23:23:14,065 DEBUG [RS:3;asf911:56613] wal.AbstractFSWAL(769): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:57095,DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d,DISK], DatanodeInfoWithStorage[127.0.0.1:53553,DS-615ebfc4-105b-4470-b106-59b6b4ae84d8,DISK], DatanodeInfoWithStorage[127.0.0.1:54793,DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e,DISK]] 2018-06-21 23:23:14,065 DEBUG [RS:4;asf911:58235] wal.AbstractFSWAL(769): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42074,DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7,DISK], DatanodeInfoWithStorage[127.0.0.1:33402,DS-d4d0c936-07f4-4962-b708-add7d68719f3,DISK], DatanodeInfoWithStorage[127.0.0.1:57095,DS-901750e1-a133-4f10-85af-4c0c301de1f4,DISK]] 2018-06-21 23:23:14,065 DEBUG [RS:0;asf911:59970] wal.AbstractFSWAL(769): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:53553,DS-615ebfc4-105b-4470-b106-59b6b4ae84d8,DISK], DatanodeInfoWithStorage[127.0.0.1:33402,DS-189d75fd-2cba-4781-8f7e-7f1fecbda947,DISK], DatanodeInfoWithStorage[127.0.0.1:57095,DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d,DISK]] 2018-06-21 23:23:14,065 DEBUG [RS:2;asf911:52507] wal.AbstractFSWAL(769): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:57095,DS-901750e1-a133-4f10-85af-4c0c301de1f4,DISK], DatanodeInfoWithStorage[127.0.0.1:53553,DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4,DISK], DatanodeInfoWithStorage[127.0.0.1:33402,DS-d4d0c936-07f4-4962-b708-add7d68719f3,DISK]] 2018-06-21 23:23:14,066 DEBUG [RS:1;asf911:40767] wal.AbstractFSWAL(769): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:33402,DS-d4d0c936-07f4-4962-b708-add7d68719f3,DISK], DatanodeInfoWithStorage[127.0.0.1:42074,DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7,DISK], DatanodeInfoWithStorage[127.0.0.1:54793,DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e,DISK]] 2018-06-21 23:23:14,066 DEBUG [M:0;asf911:55146] wal.AbstractFSWAL(769): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:54793,DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26,DISK], DatanodeInfoWithStorage[127.0.0.1:57095,DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d,DISK], DatanodeInfoWithStorage[127.0.0.1:33402,DS-189d75fd-2cba-4781-8f7e-7f1fecbda947,DISK]] 2018-06-21 23:23:14,102 INFO [PEWorker-11] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:14,151 DEBUG [PEWorker-11] zookeeper.MetaTableLocator(466): META region location doesn't exist, create it 2018-06-21 23:23:14,176 INFO [PEWorker-11] assignment.RegionTransitionProcedure(241): Dispatch pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:14,372 DEBUG [RSProcedureDispatcher-pool3-t1] master.ServerManager(746): New admin connection to asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:14,385 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-06-21 23:23:14,386 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 45549 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:14,393 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=40767] regionserver.RSRpcServices(1980): Open hbase:meta,,1.1588230740 2018-06-21 23:23:14,397 INFO [RS_OPEN_META-regionserver/asf911:0-0] wal.WALFactory(137): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-06-21 23:23:14,408 INFO [RS_OPEN_META-regionserver/asf911:0-0] wal.AbstractFSWAL(424): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C40767%2C1529623390148.meta, suffix=.meta, logDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,40767,1529623390148, archiveDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/oldWALs 2018-06-21 23:23:14,426 DEBUG [RS-EventLoopGroup-7-24] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42074,DS-85f0e371-3357-43c0-95ea-ca66d237b4a8,DISK] 2018-06-21 23:23:14,427 DEBUG [RS-EventLoopGroup-7-22] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:54793,DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26,DISK] 2018-06-21 23:23:14,426 DEBUG [RS-EventLoopGroup-7-28] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33402,DS-189d75fd-2cba-4781-8f7e-7f1fecbda947,DISK] 2018-06-21 23:23:14,459 INFO [RS_OPEN_META-regionserver/asf911:0-0] wal.AbstractFSWAL(679): New WAL /user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,40767,1529623390148/asf911.gq1.ygridcore.net%2C40767%2C1529623390148.meta.1529623394409.meta 2018-06-21 23:23:14,460 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] wal.AbstractFSWAL(769): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42074,DS-85f0e371-3357-43c0-95ea-ca66d237b4a8,DISK], DatanodeInfoWithStorage[127.0.0.1:33402,DS-189d75fd-2cba-4781-8f7e-7f1fecbda947,DISK], DatanodeInfoWithStorage[127.0.0.1:54793,DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26,DISK]] 2018-06-21 23:23:14,461 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(7094): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2018-06-21 23:23:14,491 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] coprocessor.CoprocessorHost(200): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2018-06-21 23:23:14,511 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(8072): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2018-06-21 23:23:14,517 INFO [RS_OPEN_META-regionserver/asf911:0-0] regionserver.RegionCoprocessorHost(394): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2018-06-21 23:23:14,524 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table meta 1588230740 2018-06-21 23:23:14,524 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(827): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:14,525 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(7134): checking encryption for 1588230740 2018-06-21 23:23:14,527 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(7139): checking classloading for 1588230740 2018-06-21 23:23:14,536 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/info 2018-06-21 23:23:14,536 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/info 2018-06-21 23:23:14,538 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:14,540 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:14,541 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:14,546 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/rep_barrier 2018-06-21 23:23:14,546 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/rep_barrier 2018-06-21 23:23:14,547 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:14,548 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:14,549 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:14,552 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/table 2018-06-21 23:23:14,552 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/table 2018-06-21 23:23:14,552 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:14,553 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:14,554 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:14,554 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(923): replaying wal for 1588230740 2018-06-21 23:23:14,561 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740 2018-06-21 23:23:14,561 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(931): stopping wal replay for 1588230740 2018-06-21 23:23:14,561 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(943): Cleaning up temporary data for 1588230740 2018-06-21 23:23:14,562 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(954): Cleaning up detritus for 1588230740 2018-06-21 23:23:14,565 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7M)) instead. 2018-06-21 23:23:14,566 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(976): writing seq id for 1588230740 2018-06-21 23:23:14,567 INFO [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(980): Opened 1588230740; next sequenceid=2 2018-06-21 23:23:14,567 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(987): Running coprocessor post-open hooks for 1588230740 2018-06-21 23:23:14,620 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2187): Post open deploy tasks for hbase:meta,,1.1588230740 2018-06-21 23:23:14,644 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=2, pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:14,644 DEBUG [PEWorker-13] assignment.RegionTransitionProcedure(354): Finishing pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:14,647 INFO [PEWorker-13] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:14,648 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2211): Finished post open deploy task for hbase:meta,,1.1588230740 2018-06-21 23:23:14,651 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] handler.OpenRegionHandler(128): Opened hbase:meta,,1.1588230740 on asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:14,809 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2018-06-21 23:23:15,174 WARN [DataXceiver for client DFSClient_NONMAPREDUCE_321370780_23 at /127.0.0.1:44439 [Receiving block BP-1812449274-67.195.81.155-1529623383006:blk_1073741829_1005]] datanode.BlockReceiver(422): Slow flushOrSync took 363ms (threshold=300ms), isSync:true, flushTotalNanos=5967ns 2018-06-21 23:23:15,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=55146] assignment.AssignmentManager(1002): META REPORTED: rit=OPEN, location=asf911.gq1.ygridcore.net,40767,1529623390148, table=hbase:meta, region=1588230740 2018-06-21 23:23:15,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=55146] assignment.AssignmentManager(1004): META REPORTED but no procedure found (complete?); set location=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:15,202 WARN [DataXceiver for client DFSClient_NONMAPREDUCE_321370780_23 at /127.0.0.1:44537 [Receiving block BP-1812449274-67.195.81.155-1529623383006:blk_1073741829_1005]] datanode.BlockReceiver(422): Slow flushOrSync took 390ms (threshold=300ms), isSync:true, flushTotalNanos=6376ns 2018-06-21 23:23:15,220 WARN [DataXceiver for client DFSClient_NONMAPREDUCE_321370780_23 at /127.0.0.1:60083 [Receiving block BP-1812449274-67.195.81.155-1529623383006:blk_1073741829_1005]] datanode.BlockReceiver(422): Slow flushOrSync took 408ms (threshold=300ms), isSync:true, flushTotalNanos=7063ns 2018-06-21 23:23:15,362 INFO [PEWorker-13] procedure2.ProcedureExecutor(1635): Finished subprocedure(s) of pid=1, state=RUNNABLE; InitMetaProcedure table=hbase:meta; resume parent processing. 2018-06-21 23:23:15,362 INFO [PEWorker-13] procedure2.ProcedureExecutor(1266): Finished pid=2, ppid=1, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 in 2.5690sec 2018-06-21 23:23:15,503 INFO [PEWorker-14] procedure2.ProcedureExecutor(1266): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 3.4480sec 2018-06-21 23:23:15,590 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-06-21 23:23:15,590 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 45554 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:15,676 INFO [Thread-229] master.HMaster(955): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1529623390494, completionTime=-1 2018-06-21 23:23:15,677 INFO [Thread-229] master.ServerManager(854): Finished waiting on RegionServer count=6; waited=0ms, expected min=5 server(s), max=5 server(s), master is running 2018-06-21 23:23:15,677 DEBUG [Thread-229] assignment.AssignmentManager(1210): Joining cluster... 2018-06-21 23:23:15,684 INFO [Thread-229] assignment.AssignmentManager(1221): Number of RegionServers=6 2018-06-21 23:23:15,686 INFO [Thread-229] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1529623455686 2018-06-21 23:23:15,686 INFO [Thread-229] assignment.AssignmentManager(1229): Joined the cluster in 9msec 2018-06-21 23:23:15,751 INFO [Thread-229] master.TableNamespaceManager(96): Namespace table not found. Creating... 2018-06-21 23:23:15,757 INFO [Thread-229] master.HMaster(1878): Client=null/null create 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} 2018-06-21 23:23:15,988 DEBUG [Thread-229] procedure2.ProcedureExecutor(887): Stored pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-06-21 23:23:16,119 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741837_1013{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:16,121 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741837_1013{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:16,123 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741837_1013{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:16,145 DEBUG [PEWorker-15] util.FSTableDescriptors(683): Wrote into hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2018-06-21 23:23:16,150 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(6917): creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.tmp Table name == hbase:namespace 2018-06-21 23:23:16,205 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741838_1014{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|FINALIZED]]} size 0 2018-06-21 23:23:16,207 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741838_1014{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|FINALIZED], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|FINALIZED]]} size 0 2018-06-21 23:23:16,210 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741838_1014 size 42 2018-06-21 23:23:16,211 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(827): Instantiated hbase:namespace,,1529623395752.581b5ff62f5df0e6be8aaeb23aa08c64.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:16,212 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1520): Closing 581b5ff62f5df0e6be8aaeb23aa08c64, disabling compactions & flushes 2018-06-21 23:23:16,213 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1560): Updates disabled for region hbase:namespace,,1529623395752.581b5ff62f5df0e6be8aaeb23aa08c64. 2018-06-21 23:23:16,213 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1674): Closed hbase:namespace,,1529623395752.581b5ff62f5df0e6be8aaeb23aa08c64. 2018-06-21 23:23:16,467 DEBUG [PEWorker-15] hbase.MetaTableAccessor(2153): Put {"totalColumns":2,"row":"hbase:namespace,,1529623395752.581b5ff62f5df0e6be8aaeb23aa08c64.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":1529623396396},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1529623396396}]},"ts":1529623396396} 2018-06-21 23:23:16,523 INFO [PEWorker-15] hbase.MetaTableAccessor(1528): Added 1 regions to meta. 2018-06-21 23:23:16,661 DEBUG [PEWorker-15] hbase.MetaTableAccessor(2153): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1529623396654}]},"ts":1529623396654} 2018-06-21 23:23:16,670 INFO [PEWorker-15] hbase.MetaTableAccessor(1673): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2018-06-21 23:23:16,717 DEBUG [PEWorker-15] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region hbase:namespace,,1529623395752.581b5ff62f5df0e6be8aaeb23aa08c64. 2018-06-21 23:23:16,733 INFO [PEWorker-15] procedure2.ProcedureExecutor(1516): Initialized subprocedures=[{pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=581b5ff62f5df0e6be8aaeb23aa08c64, target=asf911.gq1.ygridcore.net,58235,1529623390382}] 2018-06-21 23:23:16,808 INFO [PEWorker-15] procedure.MasterProcedureScheduler(696): pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=581b5ff62f5df0e6be8aaeb23aa08c64, target=asf911.gq1.ygridcore.net,58235,1529623390382 checking lock on 581b5ff62f5df0e6be8aaeb23aa08c64 2018-06-21 23:23:16,832 INFO [PEWorker-15] assignment.AssignProcedure(218): Starting pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=581b5ff62f5df0e6be8aaeb23aa08c64, target=asf911.gq1.ygridcore.net,58235,1529623390382; rit=OFFLINE, location=asf911.gq1.ygridcore.net,58235,1529623390382; forceNewPlan=false, retain=false 2018-06-21 23:23:16,989 INFO [master/asf911:0] balancer.BaseLoadBalancer(1497): Reassigned 1 regions. 1 retained the pre-restart assignment. 2018-06-21 23:23:16,993 INFO [PEWorker-3] assignment.RegionStateStore(198): pid=4 updating hbase:meta row=581b5ff62f5df0e6be8aaeb23aa08c64, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:17,004 INFO [PEWorker-3] assignment.RegionTransitionProcedure(241): Dispatch pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=581b5ff62f5df0e6be8aaeb23aa08c64, target=asf911.gq1.ygridcore.net,58235,1529623390382; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:17,156 DEBUG [RSProcedureDispatcher-pool3-t2] master.ServerManager(746): New admin connection to asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:17,162 INFO [RS-EventLoopGroup-7-12] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-06-21 23:23:17,163 INFO [RS-EventLoopGroup-7-12] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 42838 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:17,164 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58235] regionserver.RSRpcServices(1980): Open hbase:namespace,,1529623395752.581b5ff62f5df0e6be8aaeb23aa08c64. 2018-06-21 23:23:17,192 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf911:0-0] regionserver.HRegion(7094): Opening region: {ENCODED => 581b5ff62f5df0e6be8aaeb23aa08c64, NAME => 'hbase:namespace,,1529623395752.581b5ff62f5df0e6be8aaeb23aa08c64.', STARTKEY => '', ENDKEY => ''} 2018-06-21 23:23:17,199 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf911:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table namespace 581b5ff62f5df0e6be8aaeb23aa08c64 2018-06-21 23:23:17,199 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf911:0-0] regionserver.HRegion(827): Instantiated hbase:namespace,,1529623395752.581b5ff62f5df0e6be8aaeb23aa08c64.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:17,199 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf911:0-0] regionserver.HRegion(7134): checking encryption for 581b5ff62f5df0e6be8aaeb23aa08c64 2018-06-21 23:23:17,200 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf911:0-0] regionserver.HRegion(7139): checking classloading for 581b5ff62f5df0e6be8aaeb23aa08c64 2018-06-21 23:23:17,208 DEBUG [StoreOpener-581b5ff62f5df0e6be8aaeb23aa08c64-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/namespace/581b5ff62f5df0e6be8aaeb23aa08c64/info 2018-06-21 23:23:17,208 DEBUG [StoreOpener-581b5ff62f5df0e6be8aaeb23aa08c64-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/namespace/581b5ff62f5df0e6be8aaeb23aa08c64/info 2018-06-21 23:23:17,210 INFO [StoreOpener-581b5ff62f5df0e6be8aaeb23aa08c64-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:17,215 INFO [StoreOpener-581b5ff62f5df0e6be8aaeb23aa08c64-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:17,222 INFO [StoreOpener-581b5ff62f5df0e6be8aaeb23aa08c64-1] regionserver.HStore(327): Store=info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:17,223 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf911:0-0] regionserver.HRegion(923): replaying wal for 581b5ff62f5df0e6be8aaeb23aa08c64 2018-06-21 23:23:17,227 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf911:0-0] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/namespace/581b5ff62f5df0e6be8aaeb23aa08c64 2018-06-21 23:23:17,227 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf911:0-0] regionserver.HRegion(931): stopping wal replay for 581b5ff62f5df0e6be8aaeb23aa08c64 2018-06-21 23:23:17,227 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf911:0-0] regionserver.HRegion(943): Cleaning up temporary data for 581b5ff62f5df0e6be8aaeb23aa08c64 2018-06-21 23:23:17,229 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf911:0-0] regionserver.HRegion(954): Cleaning up detritus for 581b5ff62f5df0e6be8aaeb23aa08c64 2018-06-21 23:23:17,232 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf911:0-0] regionserver.HRegion(976): writing seq id for 581b5ff62f5df0e6be8aaeb23aa08c64 2018-06-21 23:23:17,241 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf911:0-0] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/namespace/581b5ff62f5df0e6be8aaeb23aa08c64/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-06-21 23:23:17,242 INFO [RS_OPEN_PRIORITY_REGION-regionserver/asf911:0-0] regionserver.HRegion(980): Opened 581b5ff62f5df0e6be8aaeb23aa08c64; next sequenceid=2 2018-06-21 23:23:17,242 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf911:0-0] regionserver.HRegion(987): Running coprocessor post-open hooks for 581b5ff62f5df0e6be8aaeb23aa08c64 2018-06-21 23:23:17,254 INFO [PostOpenDeployTasks:581b5ff62f5df0e6be8aaeb23aa08c64] regionserver.HRegionServer(2187): Post open deploy tasks for hbase:namespace,,1529623395752.581b5ff62f5df0e6be8aaeb23aa08c64. 2018-06-21 23:23:17,261 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=2, pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=581b5ff62f5df0e6be8aaeb23aa08c64, target=asf911.gq1.ygridcore.net,58235,1529623390382; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:17,262 DEBUG [PEWorker-4] assignment.RegionTransitionProcedure(354): Finishing pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=hbase:namespace, region=581b5ff62f5df0e6be8aaeb23aa08c64, target=asf911.gq1.ygridcore.net,58235,1529623390382; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:17,262 DEBUG [PostOpenDeployTasks:581b5ff62f5df0e6be8aaeb23aa08c64] regionserver.HRegionServer(2211): Finished post open deploy task for hbase:namespace,,1529623395752.581b5ff62f5df0e6be8aaeb23aa08c64. 2018-06-21 23:23:17,264 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/asf911:0-0] handler.OpenRegionHandler(128): Opened hbase:namespace,,1529623395752.581b5ff62f5df0e6be8aaeb23aa08c64. on asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:17,265 INFO [PEWorker-4] assignment.RegionStateStore(198): pid=4 updating hbase:meta row=581b5ff62f5df0e6be8aaeb23aa08c64, regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:17,435 INFO [PEWorker-4] procedure2.ProcedureExecutor(1635): Finished subprocedure(s) of pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace; resume parent processing. 2018-06-21 23:23:17,435 INFO [PEWorker-4] procedure2.ProcedureExecutor(1266): Finished pid=4, ppid=3, state=SUCCESS; AssignProcedure table=hbase:namespace, region=581b5ff62f5df0e6be8aaeb23aa08c64, target=asf911.gq1.ygridcore.net,58235,1529623390382 in 541msec 2018-06-21 23:23:17,436 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2153): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1529623397435}]},"ts":1529623397435} 2018-06-21 23:23:17,441 INFO [PEWorker-5] hbase.MetaTableAccessor(1673): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2018-06-21 23:23:17,541 DEBUG [Thread-229] zookeeper.ZKUtil(357): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2018-06-21 23:23:17,582 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2018-06-21 23:23:17,604 INFO [RS-EventLoopGroup-7-14] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-06-21 23:23:17,604 INFO [RS-EventLoopGroup-7-14] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 42870 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:17,732 INFO [PEWorker-5] procedure2.ProcedureExecutor(1266): Finished pid=3, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.8500sec 2018-06-21 23:23:17,839 DEBUG [Thread-229] procedure2.ProcedureExecutor(887): Stored pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-06-21 23:23:18,109 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-06-21 23:23:18,362 INFO [PEWorker-6] procedure2.ProcedureExecutor(1266): Finished pid=5, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 654msec 2018-06-21 23:23:18,558 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-06-21 23:23:18,559 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-06-21 23:23:18,570 DEBUG [Thread-229] procedure2.ProcedureExecutor(887): Stored pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-06-21 23:23:18,694 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(135): Registering adapter for the MetricRegistry: Master,sub=Coprocessor.Master.CP_org.apache.hadoop.hbase.client.TestMultiParallel$MyMasterObserver 2018-06-21 23:23:18,695 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(139): Registering Master,sub=Coprocessor.Master.CP_org.apache.hadoop.hbase.client.TestMultiParallel$MyMasterObserver Metrics about HBase MasterObservers 2018-06-21 23:23:18,696 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(135): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-06-21 23:23:18,696 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(139): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2018-06-21 23:23:18,884 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-06-21 23:23:19,115 INFO [PEWorker-7] procedure2.ProcedureExecutor(1266): Finished pid=6, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 666msec 2018-06-21 23:23:19,142 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2018-06-21 23:23:19,174 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2018-06-21 23:23:19,174 INFO [Thread-229] master.HMaster(1002): Master has completed initialization 8.620sec 2018-06-21 23:23:19,183 INFO [Thread-229] quotas.MasterQuotaManager(90): Quota support disabled 2018-06-21 23:23:19,183 INFO [Thread-229] zookeeper.ZKWatcher(205): not a secure deployment, proceeding 2018-06-21 23:23:19,203 DEBUG [Thread-229] master.HMaster(1060): Balancer post startup initialization complete, took 0 seconds 2018-06-21 23:23:19,268 INFO [Time-limited test] zookeeper.ReadOnlyZKClient(139): Connect 0x3275446e to localhost:58989 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-06-21 23:23:19,283 DEBUG [Time-limited test] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1632ac99, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-06-21 23:23:19,325 INFO [RS-EventLoopGroup-4-4] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-06-21 23:23:19,326 INFO [RS-EventLoopGroup-4-4] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 45610 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:19,342 INFO [Time-limited test] hbase.HBaseTestingUtility(1045): Minicluster is up; activeMaster=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:19,368 INFO [RS-EventLoopGroup-1-7] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-06-21 23:23:19,369 INFO [RS-EventLoopGroup-1-7] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 40574 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:19,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=55146] master.HMaster$3(1842): Client=jenkins//67.195.81.155 create 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2018-06-21 23:23:19,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=55146] procedure2.ProcedureExecutor(887): Stored pid=7, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=multi_test_table 2018-06-21 23:23:19,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=55146] master.MasterRpcServices(1142): Checking to see if procedure is done pid=7 2018-06-21 23:23:19,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=55146] master.MasterRpcServices(1142): Checking to see if procedure is done pid=7 2018-06-21 23:23:19,782 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741839_1015{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:19,784 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741839_1015{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:19,784 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741839_1015{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|FINALIZED]]} size 0 2018-06-21 23:23:19,811 DEBUG [PEWorker-8] util.FSTableDescriptors(683): Wrote into hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.tmp/data/default/multi_test_table/.tabledesc/.tableinfo.0000000001 2018-06-21 23:23:19,820 INFO [RegionOpenAndInitThread-multi_test_table-6] regionserver.HRegion(6917): creating HRegion multi_test_table HTD == 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.tmp Table name == multi_test_table 2018-06-21 23:23:19,821 INFO [RegionOpenAndInitThread-multi_test_table-3] regionserver.HRegion(6917): creating HRegion multi_test_table HTD == 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.tmp Table name == multi_test_table 2018-06-21 23:23:19,821 INFO [RegionOpenAndInitThread-multi_test_table-5] regionserver.HRegion(6917): creating HRegion multi_test_table HTD == 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.tmp Table name == multi_test_table 2018-06-21 23:23:19,821 INFO [RegionOpenAndInitThread-multi_test_table-9] regionserver.HRegion(6917): creating HRegion multi_test_table HTD == 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.tmp Table name == multi_test_table 2018-06-21 23:23:19,827 INFO [RegionOpenAndInitThread-multi_test_table-1] regionserver.HRegion(6917): creating HRegion multi_test_table HTD == 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.tmp Table name == multi_test_table 2018-06-21 23:23:19,832 INFO [RegionOpenAndInitThread-multi_test_table-14] regionserver.HRegion(6917): creating HRegion multi_test_table HTD == 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.tmp Table name == multi_test_table 2018-06-21 23:23:19,833 INFO [RegionOpenAndInitThread-multi_test_table-12] regionserver.HRegion(6917): creating HRegion multi_test_table HTD == 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.tmp Table name == multi_test_table 2018-06-21 23:23:19,835 INFO [RegionOpenAndInitThread-multi_test_table-13] regionserver.HRegion(6917): creating HRegion multi_test_table HTD == 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.tmp Table name == multi_test_table 2018-06-21 23:23:19,837 INFO [RegionOpenAndInitThread-multi_test_table-11] regionserver.HRegion(6917): creating HRegion multi_test_table HTD == 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.tmp Table name == multi_test_table 2018-06-21 23:23:19,840 INFO [RegionOpenAndInitThread-multi_test_table-4] regionserver.HRegion(6917): creating HRegion multi_test_table HTD == 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.tmp Table name == multi_test_table 2018-06-21 23:23:19,840 INFO [RegionOpenAndInitThread-multi_test_table-10] regionserver.HRegion(6917): creating HRegion multi_test_table HTD == 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.tmp Table name == multi_test_table 2018-06-21 23:23:19,840 INFO [RegionOpenAndInitThread-multi_test_table-7] regionserver.HRegion(6917): creating HRegion multi_test_table HTD == 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.tmp Table name == multi_test_table 2018-06-21 23:23:19,843 INFO [RegionOpenAndInitThread-multi_test_table-8] regionserver.HRegion(6917): creating HRegion multi_test_table HTD == 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.tmp Table name == multi_test_table 2018-06-21 23:23:19,849 INFO [RegionOpenAndInitThread-multi_test_table-15] regionserver.HRegion(6917): creating HRegion multi_test_table HTD == 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.tmp Table name == multi_test_table 2018-06-21 23:23:19,869 INFO [RegionOpenAndInitThread-multi_test_table-16] regionserver.HRegion(6917): creating HRegion multi_test_table HTD == 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.tmp Table name == multi_test_table 2018-06-21 23:23:19,876 INFO [RegionOpenAndInitThread-multi_test_table-2] regionserver.HRegion(6917): creating HRegion multi_test_table HTD == 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.tmp Table name == multi_test_table 2018-06-21 23:23:19,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=55146] master.MasterRpcServices(1142): Checking to see if procedure is done pid=7 2018-06-21 23:23:20,014 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741843_1019{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:20,024 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741843_1019 size 57 2018-06-21 23:23:20,025 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741843_1019 size 57 2018-06-21 23:23:20,026 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741846_1022{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW]]} size 0 2018-06-21 23:23:20,026 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741846_1022{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW]]} size 0 2018-06-21 23:23:20,026 DEBUG [RegionOpenAndInitThread-multi_test_table-10] regionserver.HRegion(827): Instantiated multi_test_table,jjj,1529623399386.573e96916daa12dfc083e39a253dc3a2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:20,029 DEBUG [RegionOpenAndInitThread-multi_test_table-10] regionserver.HRegion(1520): Closing 573e96916daa12dfc083e39a253dc3a2, disabling compactions & flushes 2018-06-21 23:23:20,029 DEBUG [RegionOpenAndInitThread-multi_test_table-10] regionserver.HRegion(1560): Updates disabled for region multi_test_table,jjj,1529623399386.573e96916daa12dfc083e39a253dc3a2. 2018-06-21 23:23:20,029 INFO [RegionOpenAndInitThread-multi_test_table-10] regionserver.HRegion(1674): Closed multi_test_table,jjj,1529623399386.573e96916daa12dfc083e39a253dc3a2. 2018-06-21 23:23:20,030 INFO [RegionOpenAndInitThread-multi_test_table-10] regionserver.HRegion(6917): creating HRegion multi_test_table HTD == 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.tmp Table name == multi_test_table 2018-06-21 23:23:20,033 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741846_1022{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW]]} size 0 2018-06-21 23:23:20,041 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741845_1021{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW]]} size 57 2018-06-21 23:23:20,041 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741844_1020{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW]]} size 57 2018-06-21 23:23:20,042 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741845_1021 size 57 2018-06-21 23:23:20,042 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741842_1018{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW]]} size 57 2018-06-21 23:23:20,042 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741842_1018 size 57 2018-06-21 23:23:20,043 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741841_1017{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:20,043 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741847_1023{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW]]} size 0 2018-06-21 23:23:20,043 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741845_1021 size 57 2018-06-21 23:23:20,043 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741844_1020 size 57 2018-06-21 23:23:20,043 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741842_1018 size 57 2018-06-21 23:23:20,047 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741841_1017{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:20,047 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741847_1023{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW]]} size 0 2018-06-21 23:23:20,048 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741844_1020 size 57 2018-06-21 23:23:20,054 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-06-21 23:23:20,061 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741847_1023{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW]]} size 0 2018-06-21 23:23:20,062 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741841_1017{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:20,074 DEBUG [RegionOpenAndInitThread-multi_test_table-8] regionserver.HRegion(827): Instantiated multi_test_table,hhh,1529623399386.91ab16f39c84b448bdc2e5a1e1fc801b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:20,074 DEBUG [RegionOpenAndInitThread-multi_test_table-8] regionserver.HRegion(1520): Closing 91ab16f39c84b448bdc2e5a1e1fc801b, disabling compactions & flushes 2018-06-21 23:23:20,074 DEBUG [RegionOpenAndInitThread-multi_test_table-8] regionserver.HRegion(1560): Updates disabled for region multi_test_table,hhh,1529623399386.91ab16f39c84b448bdc2e5a1e1fc801b. 2018-06-21 23:23:20,082 INFO [RegionOpenAndInitThread-multi_test_table-8] regionserver.HRegion(1674): Closed multi_test_table,hhh,1529623399386.91ab16f39c84b448bdc2e5a1e1fc801b. 2018-06-21 23:23:20,082 INFO [RegionOpenAndInitThread-multi_test_table-8] regionserver.HRegion(6917): creating HRegion multi_test_table HTD == 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.tmp Table name == multi_test_table 2018-06-21 23:23:20,084 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741840_1016{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:20,084 DEBUG [RegionOpenAndInitThread-multi_test_table-3] regionserver.HRegion(827): Instantiated multi_test_table,ccc,1529623399386.124da0ae361b264a91ea989bcd12b459.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:20,084 DEBUG [RegionOpenAndInitThread-multi_test_table-3] regionserver.HRegion(1520): Closing 124da0ae361b264a91ea989bcd12b459, disabling compactions & flushes 2018-06-21 23:23:20,085 DEBUG [RegionOpenAndInitThread-multi_test_table-3] regionserver.HRegion(1560): Updates disabled for region multi_test_table,ccc,1529623399386.124da0ae361b264a91ea989bcd12b459. 2018-06-21 23:23:20,085 INFO [RegionOpenAndInitThread-multi_test_table-3] regionserver.HRegion(1674): Closed multi_test_table,ccc,1529623399386.124da0ae361b264a91ea989bcd12b459. 2018-06-21 23:23:20,085 INFO [RegionOpenAndInitThread-multi_test_table-3] regionserver.HRegion(6917): creating HRegion multi_test_table HTD == 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.tmp Table name == multi_test_table 2018-06-21 23:23:20,099 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741854_1030{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|FINALIZED]]} size 0 2018-06-21 23:23:20,100 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741849_1025{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|FINALIZED]]} size 0 2018-06-21 23:23:20,100 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741840_1016{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:20,100 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741854_1030{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|FINALIZED]]} size 0 2018-06-21 23:23:20,100 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741840_1016{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:20,101 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741854_1030{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|FINALIZED]]} size 0 2018-06-21 23:23:20,101 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741848_1024{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW]]} size 0 2018-06-21 23:23:20,101 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741853_1029{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:20,102 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741849_1025{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|FINALIZED]]} size 0 2018-06-21 23:23:20,102 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741853_1029{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:20,102 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741848_1024{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW]]} size 0 2018-06-21 23:23:20,103 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741849_1025{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|FINALIZED]]} size 0 2018-06-21 23:23:20,103 DEBUG [RegionOpenAndInitThread-multi_test_table-9] regionserver.HRegion(827): Instantiated multi_test_table,iii,1529623399386.04d71925444c7c823938ba3d9ad0504d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:20,103 DEBUG [RegionOpenAndInitThread-multi_test_table-9] regionserver.HRegion(1520): Closing 04d71925444c7c823938ba3d9ad0504d, disabling compactions & flushes 2018-06-21 23:23:20,103 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741853_1029{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:20,104 DEBUG [RegionOpenAndInitThread-multi_test_table-9] regionserver.HRegion(1560): Updates disabled for region multi_test_table,iii,1529623399386.04d71925444c7c823938ba3d9ad0504d. 2018-06-21 23:23:20,104 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741850_1026{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW]]} size 0 2018-06-21 23:23:20,104 INFO [RegionOpenAndInitThread-multi_test_table-9] regionserver.HRegion(1674): Closed multi_test_table,iii,1529623399386.04d71925444c7c823938ba3d9ad0504d. 2018-06-21 23:23:20,104 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741850_1026{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW]]} size 0 2018-06-21 23:23:20,111 INFO [RegionOpenAndInitThread-multi_test_table-9] regionserver.HRegion(6917): creating HRegion multi_test_table HTD == 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.tmp Table name == multi_test_table 2018-06-21 23:23:20,114 DEBUG [RegionOpenAndInitThread-multi_test_table-6] regionserver.HRegion(827): Instantiated multi_test_table,fff,1529623399386.da9d0eefced1af8dfad2324a5d9e1bf7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:20,115 DEBUG [RegionOpenAndInitThread-multi_test_table-6] regionserver.HRegion(1520): Closing da9d0eefced1af8dfad2324a5d9e1bf7, disabling compactions & flushes 2018-06-21 23:23:20,115 DEBUG [RegionOpenAndInitThread-multi_test_table-6] regionserver.HRegion(1560): Updates disabled for region multi_test_table,fff,1529623399386.da9d0eefced1af8dfad2324a5d9e1bf7. 2018-06-21 23:23:20,115 INFO [RegionOpenAndInitThread-multi_test_table-6] regionserver.HRegion(1674): Closed multi_test_table,fff,1529623399386.da9d0eefced1af8dfad2324a5d9e1bf7. 2018-06-21 23:23:20,116 INFO [RegionOpenAndInitThread-multi_test_table-6] regionserver.HRegion(6917): creating HRegion multi_test_table HTD == 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.tmp Table name == multi_test_table 2018-06-21 23:23:20,116 DEBUG [RegionOpenAndInitThread-multi_test_table-16] regionserver.HRegion(827): Instantiated multi_test_table,ppp,1529623399386.36d0a953419e9f81636aebf73e7ed5df.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:20,116 DEBUG [RegionOpenAndInitThread-multi_test_table-1] regionserver.HRegion(827): Instantiated multi_test_table,,1529623399386.cb0df54f72fae3cd39a03514dbbaa19c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:20,116 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741850_1026 size 57 2018-06-21 23:23:20,116 DEBUG [RegionOpenAndInitThread-multi_test_table-16] regionserver.HRegion(1520): Closing 36d0a953419e9f81636aebf73e7ed5df, disabling compactions & flushes 2018-06-21 23:23:20,118 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741848_1024 size 57 2018-06-21 23:23:20,116 DEBUG [RegionOpenAndInitThread-multi_test_table-1] regionserver.HRegion(1520): Closing cb0df54f72fae3cd39a03514dbbaa19c, disabling compactions & flushes 2018-06-21 23:23:20,120 DEBUG [RegionOpenAndInitThread-multi_test_table-1] regionserver.HRegion(1560): Updates disabled for region multi_test_table,,1529623399386.cb0df54f72fae3cd39a03514dbbaa19c. 2018-06-21 23:23:20,120 INFO [RegionOpenAndInitThread-multi_test_table-1] regionserver.HRegion(1674): Closed multi_test_table,,1529623399386.cb0df54f72fae3cd39a03514dbbaa19c. 2018-06-21 23:23:20,120 DEBUG [RegionOpenAndInitThread-multi_test_table-4] regionserver.HRegion(827): Instantiated multi_test_table,ddd,1529623399386.e66f718b26d1c4420497ccfb42f08321.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:20,127 INFO [RegionOpenAndInitThread-multi_test_table-1] regionserver.HRegion(6917): creating HRegion multi_test_table HTD == 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.tmp Table name == multi_test_table 2018-06-21 23:23:20,120 DEBUG [RegionOpenAndInitThread-multi_test_table-13] regionserver.HRegion(827): Instantiated multi_test_table,mmm,1529623399386.28b5f974b61112896f4c4f1127dc2b99.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:20,128 DEBUG [RegionOpenAndInitThread-multi_test_table-13] regionserver.HRegion(1520): Closing 28b5f974b61112896f4c4f1127dc2b99, disabling compactions & flushes 2018-06-21 23:23:20,129 DEBUG [RegionOpenAndInitThread-multi_test_table-13] regionserver.HRegion(1560): Updates disabled for region multi_test_table,mmm,1529623399386.28b5f974b61112896f4c4f1127dc2b99. 2018-06-21 23:23:20,131 INFO [RegionOpenAndInitThread-multi_test_table-13] regionserver.HRegion(1674): Closed multi_test_table,mmm,1529623399386.28b5f974b61112896f4c4f1127dc2b99. 2018-06-21 23:23:20,120 DEBUG [RegionOpenAndInitThread-multi_test_table-16] regionserver.HRegion(1560): Updates disabled for region multi_test_table,ppp,1529623399386.36d0a953419e9f81636aebf73e7ed5df. 2018-06-21 23:23:20,131 INFO [RegionOpenAndInitThread-multi_test_table-16] regionserver.HRegion(1674): Closed multi_test_table,ppp,1529623399386.36d0a953419e9f81636aebf73e7ed5df. 2018-06-21 23:23:20,127 DEBUG [RegionOpenAndInitThread-multi_test_table-11] regionserver.HRegion(827): Instantiated multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:20,127 DEBUG [RegionOpenAndInitThread-multi_test_table-4] regionserver.HRegion(1520): Closing e66f718b26d1c4420497ccfb42f08321, disabling compactions & flushes 2018-06-21 23:23:20,120 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741852_1028{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:20,132 INFO [RegionOpenAndInitThread-multi_test_table-13] regionserver.HRegion(6917): creating HRegion multi_test_table HTD == 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.tmp Table name == multi_test_table 2018-06-21 23:23:20,136 DEBUG [RegionOpenAndInitThread-multi_test_table-4] regionserver.HRegion(1560): Updates disabled for region multi_test_table,ddd,1529623399386.e66f718b26d1c4420497ccfb42f08321. 2018-06-21 23:23:20,136 INFO [RegionOpenAndInitThread-multi_test_table-4] regionserver.HRegion(1674): Closed multi_test_table,ddd,1529623399386.e66f718b26d1c4420497ccfb42f08321. 2018-06-21 23:23:20,137 DEBUG [RegionOpenAndInitThread-multi_test_table-11] regionserver.HRegion(1520): Closing fee67ae5d4f2b2be35fa516877a8d173, disabling compactions & flushes 2018-06-21 23:23:20,137 DEBUG [RegionOpenAndInitThread-multi_test_table-11] regionserver.HRegion(1560): Updates disabled for region multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173. 2018-06-21 23:23:20,140 INFO [RegionOpenAndInitThread-multi_test_table-4] regionserver.HRegion(6917): creating HRegion multi_test_table HTD == 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.tmp Table name == multi_test_table 2018-06-21 23:23:20,142 INFO [RegionOpenAndInitThread-multi_test_table-16] regionserver.HRegion(6917): creating HRegion multi_test_table HTD == 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.tmp Table name == multi_test_table 2018-06-21 23:23:20,142 INFO [RegionOpenAndInitThread-multi_test_table-11] regionserver.HRegion(1674): Closed multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173. 2018-06-21 23:23:20,142 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741851_1027{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW]]} size 57 2018-06-21 23:23:20,142 DEBUG [RegionOpenAndInitThread-multi_test_table-15] regionserver.HRegion(827): Instantiated multi_test_table,ooo,1529623399386.1449fc3d526b1c6ab2fd158edec23277.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:20,143 DEBUG [RegionOpenAndInitThread-multi_test_table-15] regionserver.HRegion(1520): Closing 1449fc3d526b1c6ab2fd158edec23277, disabling compactions & flushes 2018-06-21 23:23:20,143 DEBUG [RegionOpenAndInitThread-multi_test_table-15] regionserver.HRegion(1560): Updates disabled for region multi_test_table,ooo,1529623399386.1449fc3d526b1c6ab2fd158edec23277. 2018-06-21 23:23:20,143 INFO [RegionOpenAndInitThread-multi_test_table-11] regionserver.HRegion(6917): creating HRegion multi_test_table HTD == 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.tmp Table name == multi_test_table 2018-06-21 23:23:20,143 INFO [RegionOpenAndInitThread-multi_test_table-15] regionserver.HRegion(1674): Closed multi_test_table,ooo,1529623399386.1449fc3d526b1c6ab2fd158edec23277. 2018-06-21 23:23:20,143 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741851_1027 size 57 2018-06-21 23:23:20,144 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741852_1028 size 57 2018-06-21 23:23:20,144 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741852_1028 size 57 2018-06-21 23:23:20,144 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741851_1027 size 57 2018-06-21 23:23:20,145 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741855_1031{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW]]} size 57 2018-06-21 23:23:20,145 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741855_1031 size 57 2018-06-21 23:23:20,145 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741855_1031 size 57 2018-06-21 23:23:20,212 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741857_1033{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|FINALIZED]]} size 0 2018-06-21 23:23:20,213 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741856_1032{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:20,213 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741858_1034{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|FINALIZED]]} size 0 2018-06-21 23:23:20,214 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741856_1032{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW]]} size 0 2018-06-21 23:23:20,216 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741858_1034{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|FINALIZED]]} size 0 2018-06-21 23:23:20,216 DEBUG [RegionOpenAndInitThread-multi_test_table-10] regionserver.HRegion(827): Instantiated multi_test_table,qqq,1529623399386.7480a111a1f600aa34b97a77e47640bc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:20,217 DEBUG [RegionOpenAndInitThread-multi_test_table-10] regionserver.HRegion(1520): Closing 7480a111a1f600aa34b97a77e47640bc, disabling compactions & flushes 2018-06-21 23:23:20,217 DEBUG [RegionOpenAndInitThread-multi_test_table-10] regionserver.HRegion(1560): Updates disabled for region multi_test_table,qqq,1529623399386.7480a111a1f600aa34b97a77e47640bc. 2018-06-21 23:23:20,217 INFO [RegionOpenAndInitThread-multi_test_table-10] regionserver.HRegion(1674): Closed multi_test_table,qqq,1529623399386.7480a111a1f600aa34b97a77e47640bc. 2018-06-21 23:23:20,218 DEBUG [RegionOpenAndInitThread-multi_test_table-8] regionserver.HRegion(827): Instantiated multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:20,218 DEBUG [RegionOpenAndInitThread-multi_test_table-8] regionserver.HRegion(1520): Closing 8e2537011449fca6295a357cbde76e6c, disabling compactions & flushes 2018-06-21 23:23:20,218 DEBUG [RegionOpenAndInitThread-multi_test_table-8] regionserver.HRegion(1560): Updates disabled for region multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c. 2018-06-21 23:23:20,218 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741857_1033 size 57 2018-06-21 23:23:20,218 INFO [RegionOpenAndInitThread-multi_test_table-8] regionserver.HRegion(1674): Closed multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c. 2018-06-21 23:23:20,218 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741856_1032 size 57 2018-06-21 23:23:20,218 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741857_1033 size 57 2018-06-21 23:23:20,219 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741858_1034{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|FINALIZED]]} size 0 2018-06-21 23:23:20,227 DEBUG [RegionOpenAndInitThread-multi_test_table-3] regionserver.HRegion(827): Instantiated multi_test_table,sss,1529623399386.d4ac3b44d02d68507d6317e201672011.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:20,227 DEBUG [RegionOpenAndInitThread-multi_test_table-3] regionserver.HRegion(1520): Closing d4ac3b44d02d68507d6317e201672011, disabling compactions & flushes 2018-06-21 23:23:20,228 DEBUG [RegionOpenAndInitThread-multi_test_table-3] regionserver.HRegion(1560): Updates disabled for region multi_test_table,sss,1529623399386.d4ac3b44d02d68507d6317e201672011. 2018-06-21 23:23:20,228 INFO [RegionOpenAndInitThread-multi_test_table-3] regionserver.HRegion(1674): Closed multi_test_table,sss,1529623399386.d4ac3b44d02d68507d6317e201672011. 2018-06-21 23:23:20,271 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741859_1035{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|FINALIZED]]} size 0 2018-06-21 23:23:20,275 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741859_1035 size 57 2018-06-21 23:23:20,282 DEBUG [RegionOpenAndInitThread-multi_test_table-9] regionserver.HRegion(827): Instantiated multi_test_table,ttt,1529623399386.dff6ab063ea78552b5e3c3ae7760ca5a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:20,282 DEBUG [RegionOpenAndInitThread-multi_test_table-9] regionserver.HRegion(1520): Closing dff6ab063ea78552b5e3c3ae7760ca5a, disabling compactions & flushes 2018-06-21 23:23:20,282 DEBUG [RegionOpenAndInitThread-multi_test_table-9] regionserver.HRegion(1560): Updates disabled for region multi_test_table,ttt,1529623399386.dff6ab063ea78552b5e3c3ae7760ca5a. 2018-06-21 23:23:20,283 INFO [RegionOpenAndInitThread-multi_test_table-9] regionserver.HRegion(1674): Closed multi_test_table,ttt,1529623399386.dff6ab063ea78552b5e3c3ae7760ca5a. 2018-06-21 23:23:20,283 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741859_1035 size 57 2018-06-21 23:23:20,285 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741861_1037{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW]]} size 57 2018-06-21 23:23:20,287 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741860_1036{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW]]} size 57 2018-06-21 23:23:20,287 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741860_1036 size 57 2018-06-21 23:23:20,288 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741860_1036 size 57 2018-06-21 23:23:20,288 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741861_1037 size 57 2018-06-21 23:23:20,288 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741861_1037 size 57 2018-06-21 23:23:20,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=55146] master.MasterRpcServices(1142): Checking to see if procedure is done pid=7 2018-06-21 23:23:20,301 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741863_1039{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW]]} size 0 2018-06-21 23:23:20,302 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741863_1039{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW]]} size 0 2018-06-21 23:23:20,302 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741863_1039{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW]]} size 0 2018-06-21 23:23:20,302 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741865_1041{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW]]} size 0 2018-06-21 23:23:20,304 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741864_1040{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW]]} size 0 2018-06-21 23:23:20,304 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741862_1038{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:20,304 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741862_1038{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:20,304 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741864_1040{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW]]} size 0 2018-06-21 23:23:20,304 DEBUG [RegionOpenAndInitThread-multi_test_table-16] regionserver.HRegion(827): Instantiated multi_test_table,xxx,1529623399386.77a32d153e5d8148ca9d599c6a3cb695.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:20,305 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741865_1041{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW]]} size 0 2018-06-21 23:23:20,305 DEBUG [RegionOpenAndInitThread-multi_test_table-16] regionserver.HRegion(1520): Closing 77a32d153e5d8148ca9d599c6a3cb695, disabling compactions & flushes 2018-06-21 23:23:20,305 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741862_1038{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:20,305 DEBUG [RegionOpenAndInitThread-multi_test_table-16] regionserver.HRegion(1560): Updates disabled for region multi_test_table,xxx,1529623399386.77a32d153e5d8148ca9d599c6a3cb695. 2018-06-21 23:23:20,305 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741864_1040{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW]]} size 0 2018-06-21 23:23:20,305 INFO [RegionOpenAndInitThread-multi_test_table-16] regionserver.HRegion(1674): Closed multi_test_table,xxx,1529623399386.77a32d153e5d8148ca9d599c6a3cb695. 2018-06-21 23:23:20,306 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741865_1041{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW]]} size 0 2018-06-21 23:23:20,312 DEBUG [RegionOpenAndInitThread-multi_test_table-4] regionserver.HRegion(827): Instantiated multi_test_table,yyy,1529623399386.7006d69958189f4e3b610c07f795d5d5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:20,312 DEBUG [RegionOpenAndInitThread-multi_test_table-11] regionserver.HRegion(827): Instantiated multi_test_table,zzz,1529623399386.794028a2b8fc0a516890763306629d4c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:20,312 DEBUG [RegionOpenAndInitThread-multi_test_table-4] regionserver.HRegion(1520): Closing 7006d69958189f4e3b610c07f795d5d5, disabling compactions & flushes 2018-06-21 23:23:20,313 DEBUG [RegionOpenAndInitThread-multi_test_table-13] regionserver.HRegion(827): Instantiated multi_test_table,www,1529623399386.a657e5e90b9697e4cf81bc4bbde44800.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:20,313 DEBUG [RegionOpenAndInitThread-multi_test_table-11] regionserver.HRegion(1520): Closing 794028a2b8fc0a516890763306629d4c, disabling compactions & flushes 2018-06-21 23:23:20,313 DEBUG [RegionOpenAndInitThread-multi_test_table-13] regionserver.HRegion(1520): Closing a657e5e90b9697e4cf81bc4bbde44800, disabling compactions & flushes 2018-06-21 23:23:20,313 DEBUG [RegionOpenAndInitThread-multi_test_table-4] regionserver.HRegion(1560): Updates disabled for region multi_test_table,yyy,1529623399386.7006d69958189f4e3b610c07f795d5d5. 2018-06-21 23:23:20,313 DEBUG [RegionOpenAndInitThread-multi_test_table-13] regionserver.HRegion(1560): Updates disabled for region multi_test_table,www,1529623399386.a657e5e90b9697e4cf81bc4bbde44800. 2018-06-21 23:23:20,313 DEBUG [RegionOpenAndInitThread-multi_test_table-11] regionserver.HRegion(1560): Updates disabled for region multi_test_table,zzz,1529623399386.794028a2b8fc0a516890763306629d4c. 2018-06-21 23:23:20,313 INFO [RegionOpenAndInitThread-multi_test_table-13] regionserver.HRegion(1674): Closed multi_test_table,www,1529623399386.a657e5e90b9697e4cf81bc4bbde44800. 2018-06-21 23:23:20,313 INFO [RegionOpenAndInitThread-multi_test_table-4] regionserver.HRegion(1674): Closed multi_test_table,yyy,1529623399386.7006d69958189f4e3b610c07f795d5d5. 2018-06-21 23:23:20,313 INFO [RegionOpenAndInitThread-multi_test_table-11] regionserver.HRegion(1674): Closed multi_test_table,zzz,1529623399386.794028a2b8fc0a516890763306629d4c. 2018-06-21 23:23:20,450 DEBUG [RegionOpenAndInitThread-multi_test_table-5] regionserver.HRegion(827): Instantiated multi_test_table,eee,1529623399386.c793c2c3948aad84865d30ebf71a529e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:20,450 DEBUG [RegionOpenAndInitThread-multi_test_table-14] regionserver.HRegion(827): Instantiated multi_test_table,nnn,1529623399386.b1a8d69ffdd43e80f30a919effdc7c4e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:20,450 DEBUG [RegionOpenAndInitThread-multi_test_table-12] regionserver.HRegion(827): Instantiated multi_test_table,lll,1529623399386.77885b472341ce945c76aa06a4627146.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:20,450 DEBUG [RegionOpenAndInitThread-multi_test_table-5] regionserver.HRegion(1520): Closing c793c2c3948aad84865d30ebf71a529e, disabling compactions & flushes 2018-06-21 23:23:20,451 DEBUG [RegionOpenAndInitThread-multi_test_table-5] regionserver.HRegion(1560): Updates disabled for region multi_test_table,eee,1529623399386.c793c2c3948aad84865d30ebf71a529e. 2018-06-21 23:23:20,451 INFO [RegionOpenAndInitThread-multi_test_table-5] regionserver.HRegion(1674): Closed multi_test_table,eee,1529623399386.c793c2c3948aad84865d30ebf71a529e. 2018-06-21 23:23:20,450 DEBUG [RegionOpenAndInitThread-multi_test_table-12] regionserver.HRegion(1520): Closing 77885b472341ce945c76aa06a4627146, disabling compactions & flushes 2018-06-21 23:23:20,450 DEBUG [RegionOpenAndInitThread-multi_test_table-14] regionserver.HRegion(1520): Closing b1a8d69ffdd43e80f30a919effdc7c4e, disabling compactions & flushes 2018-06-21 23:23:20,451 DEBUG [RegionOpenAndInitThread-multi_test_table-12] regionserver.HRegion(1560): Updates disabled for region multi_test_table,lll,1529623399386.77885b472341ce945c76aa06a4627146. 2018-06-21 23:23:20,451 DEBUG [RegionOpenAndInitThread-multi_test_table-14] regionserver.HRegion(1560): Updates disabled for region multi_test_table,nnn,1529623399386.b1a8d69ffdd43e80f30a919effdc7c4e. 2018-06-21 23:23:20,451 INFO [RegionOpenAndInitThread-multi_test_table-12] regionserver.HRegion(1674): Closed multi_test_table,lll,1529623399386.77885b472341ce945c76aa06a4627146. 2018-06-21 23:23:20,452 INFO [RegionOpenAndInitThread-multi_test_table-14] regionserver.HRegion(1674): Closed multi_test_table,nnn,1529623399386.b1a8d69ffdd43e80f30a919effdc7c4e. 2018-06-21 23:23:20,549 DEBUG [RegionOpenAndInitThread-multi_test_table-7] regionserver.HRegion(827): Instantiated multi_test_table,ggg,1529623399386.10cdebbb720d98123807841a0febf405.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:20,550 DEBUG [RegionOpenAndInitThread-multi_test_table-7] regionserver.HRegion(1520): Closing 10cdebbb720d98123807841a0febf405, disabling compactions & flushes 2018-06-21 23:23:20,550 DEBUG [RegionOpenAndInitThread-multi_test_table-7] regionserver.HRegion(1560): Updates disabled for region multi_test_table,ggg,1529623399386.10cdebbb720d98123807841a0febf405. 2018-06-21 23:23:20,552 INFO [RegionOpenAndInitThread-multi_test_table-7] regionserver.HRegion(1674): Closed multi_test_table,ggg,1529623399386.10cdebbb720d98123807841a0febf405. 2018-06-21 23:23:20,552 DEBUG [RegionOpenAndInitThread-multi_test_table-2] regionserver.HRegion(827): Instantiated multi_test_table,bbb,1529623399386.e04a2e293b4c1bbd21e87d569c11154e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:20,552 DEBUG [RegionOpenAndInitThread-multi_test_table-2] regionserver.HRegion(1520): Closing e04a2e293b4c1bbd21e87d569c11154e, disabling compactions & flushes 2018-06-21 23:23:20,552 DEBUG [RegionOpenAndInitThread-multi_test_table-2] regionserver.HRegion(1560): Updates disabled for region multi_test_table,bbb,1529623399386.e04a2e293b4c1bbd21e87d569c11154e. 2018-06-21 23:23:20,553 INFO [RegionOpenAndInitThread-multi_test_table-2] regionserver.HRegion(1674): Closed multi_test_table,bbb,1529623399386.e04a2e293b4c1bbd21e87d569c11154e. 2018-06-21 23:23:20,688 DEBUG [RegionOpenAndInitThread-multi_test_table-1] regionserver.HRegion(827): Instantiated multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:20,688 DEBUG [RegionOpenAndInitThread-multi_test_table-6] regionserver.HRegion(827): Instantiated multi_test_table,uuu,1529623399386.40503e5e9a757a4ccf579bf29b8d78c6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:20,689 DEBUG [RegionOpenAndInitThread-multi_test_table-1] regionserver.HRegion(1520): Closing 0dfb3ebb8c759d632ec793ac0387ec95, disabling compactions & flushes 2018-06-21 23:23:20,689 DEBUG [RegionOpenAndInitThread-multi_test_table-1] regionserver.HRegion(1560): Updates disabled for region multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95. 2018-06-21 23:23:20,689 INFO [RegionOpenAndInitThread-multi_test_table-1] regionserver.HRegion(1674): Closed multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95. 2018-06-21 23:23:20,689 DEBUG [RegionOpenAndInitThread-multi_test_table-6] regionserver.HRegion(1520): Closing 40503e5e9a757a4ccf579bf29b8d78c6, disabling compactions & flushes 2018-06-21 23:23:20,689 DEBUG [RegionOpenAndInitThread-multi_test_table-6] regionserver.HRegion(1560): Updates disabled for region multi_test_table,uuu,1529623399386.40503e5e9a757a4ccf579bf29b8d78c6. 2018-06-21 23:23:20,690 INFO [RegionOpenAndInitThread-multi_test_table-6] regionserver.HRegion(1674): Closed multi_test_table,uuu,1529623399386.40503e5e9a757a4ccf579bf29b8d78c6. 2018-06-21 23:23:20,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=55146] master.MasterRpcServices(1142): Checking to see if procedure is done pid=7 2018-06-21 23:23:20,823 DEBUG [PEWorker-8] hbase.MetaTableAccessor(2153): Put {"totalColumns":2,"row":"multi_test_table,jjj,1529623399386.573e96916daa12dfc083e39a253dc3a2.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":1529623400821},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1529623400821}]},"ts":1529623400821} 2018-06-21 23:23:20,824 DEBUG [PEWorker-8] hbase.MetaTableAccessor(2153): Put {"totalColumns":2,"row":"multi_test_table,hhh,1529623399386.91ab16f39c84b448bdc2e5a1e1fc801b.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":1529623400821},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1529623400821}]},"ts":1529623400821} 2018-06-21 23:23:20,824 DEBUG [PEWorker-8] hbase.MetaTableAccessor(2153): Put {"totalColumns":2,"row":"multi_test_table,ccc,1529623399386.124da0ae361b264a91ea989bcd12b459.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":1529623400821},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1529623400821}]},"ts":1529623400821} 2018-06-21 23:23:20,824 DEBUG [PEWorker-8] hbase.MetaTableAccessor(2153): Put {"totalColumns":2,"row":"multi_test_table,iii,1529623399386.04d71925444c7c823938ba3d9ad0504d.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":1529623400821},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1529623400821}]},"ts":1529623400821} 2018-06-21 23:23:20,825 DEBUG [PEWorker-8] hbase.MetaTableAccessor(2153): Put {"totalColumns":2,"row":"multi_test_table,fff,1529623399386.da9d0eefced1af8dfad2324a5d9e1bf7.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":1529623400821},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1529623400821}]},"ts":1529623400821} 2018-06-21 23:23:20,825 DEBUG [PEWorker-8] hbase.MetaTableAccessor(2153): Put {"totalColumns":2,"row":"multi_test_table,,1529623399386.cb0df54f72fae3cd39a03514dbbaa19c.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":1529623400821},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1529623400821}]},"ts":1529623400821} 2018-06-21 23:23:20,825 DEBUG [PEWorker-8] hbase.MetaTableAccessor(2153): Put {"totalColumns":2,"row":"multi_test_table,mmm,1529623399386.28b5f974b61112896f4c4f1127dc2b99.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":1529623400821},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1529623400821}]},"ts":1529623400821} 2018-06-21 23:23:20,826 DEBUG [PEWorker-8] hbase.MetaTableAccessor(2153): Put {"totalColumns":2,"row":"multi_test_table,ppp,1529623399386.36d0a953419e9f81636aebf73e7ed5df.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":1529623400821},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1529623400821}]},"ts":1529623400821} 2018-06-21 23:23:20,826 DEBUG [PEWorker-8] hbase.MetaTableAccessor(2153): Put {"totalColumns":2,"row":"multi_test_table,ddd,1529623399386.e66f718b26d1c4420497ccfb42f08321.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":1529623400821},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1529623400821}]},"ts":1529623400821} 2018-06-21 23:23:20,826 DEBUG [PEWorker-8] hbase.MetaTableAccessor(2153): Put {"totalColumns":2,"row":"multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":1529623400821},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1529623400821}]},"ts":1529623400821} 2018-06-21 23:23:20,827 DEBUG [PEWorker-8] hbase.MetaTableAccessor(2153): Put {"totalColumns":2,"row":"multi_test_table,ooo,1529623399386.1449fc3d526b1c6ab2fd158edec23277.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":1529623400821},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1529623400821}]},"ts":1529623400821} 2018-06-21 23:23:20,827 DEBUG [PEWorker-8] hbase.MetaTableAccessor(2153): Put {"totalColumns":2,"row":"multi_test_table,qqq,1529623399386.7480a111a1f600aa34b97a77e47640bc.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":1529623400821},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1529623400821}]},"ts":1529623400821} 2018-06-21 23:23:20,827 DEBUG [PEWorker-8] hbase.MetaTableAccessor(2153): Put {"totalColumns":2,"row":"multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":1529623400821},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1529623400821}]},"ts":1529623400821} 2018-06-21 23:23:20,828 DEBUG [PEWorker-8] hbase.MetaTableAccessor(2153): Put {"totalColumns":2,"row":"multi_test_table,sss,1529623399386.d4ac3b44d02d68507d6317e201672011.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":1529623400821},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1529623400821}]},"ts":1529623400821} 2018-06-21 23:23:20,828 DEBUG [PEWorker-8] hbase.MetaTableAccessor(2153): Put {"totalColumns":2,"row":"multi_test_table,ttt,1529623399386.dff6ab063ea78552b5e3c3ae7760ca5a.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":1529623400821},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1529623400821}]},"ts":1529623400821} 2018-06-21 23:23:20,828 DEBUG [PEWorker-8] hbase.MetaTableAccessor(2153): Put {"totalColumns":2,"row":"multi_test_table,xxx,1529623399386.77a32d153e5d8148ca9d599c6a3cb695.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":1529623400821},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1529623400821}]},"ts":1529623400821} 2018-06-21 23:23:20,828 DEBUG [PEWorker-8] hbase.MetaTableAccessor(2153): Put {"totalColumns":2,"row":"multi_test_table,www,1529623399386.a657e5e90b9697e4cf81bc4bbde44800.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":1529623400821},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1529623400821}]},"ts":1529623400821} 2018-06-21 23:23:20,829 DEBUG [PEWorker-8] hbase.MetaTableAccessor(2153): Put {"totalColumns":2,"row":"multi_test_table,yyy,1529623399386.7006d69958189f4e3b610c07f795d5d5.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":1529623400821},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1529623400821}]},"ts":1529623400821} 2018-06-21 23:23:20,829 DEBUG [PEWorker-8] hbase.MetaTableAccessor(2153): Put {"totalColumns":2,"row":"multi_test_table,zzz,1529623399386.794028a2b8fc0a516890763306629d4c.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":1529623400821},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1529623400821}]},"ts":1529623400821} 2018-06-21 23:23:20,829 DEBUG [PEWorker-8] hbase.MetaTableAccessor(2153): Put {"totalColumns":2,"row":"multi_test_table,eee,1529623399386.c793c2c3948aad84865d30ebf71a529e.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":1529623400821},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1529623400821}]},"ts":1529623400821} 2018-06-21 23:23:20,830 DEBUG [PEWorker-8] hbase.MetaTableAccessor(2153): Put {"totalColumns":2,"row":"multi_test_table,lll,1529623399386.77885b472341ce945c76aa06a4627146.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":1529623400821},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1529623400821}]},"ts":1529623400821} 2018-06-21 23:23:20,830 DEBUG [PEWorker-8] hbase.MetaTableAccessor(2153): Put {"totalColumns":2,"row":"multi_test_table,nnn,1529623399386.b1a8d69ffdd43e80f30a919effdc7c4e.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":1529623400821},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1529623400821}]},"ts":1529623400821} 2018-06-21 23:23:20,830 DEBUG [PEWorker-8] hbase.MetaTableAccessor(2153): Put {"totalColumns":2,"row":"multi_test_table,ggg,1529623399386.10cdebbb720d98123807841a0febf405.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":1529623400821},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1529623400821}]},"ts":1529623400821} 2018-06-21 23:23:20,830 DEBUG [PEWorker-8] hbase.MetaTableAccessor(2153): Put {"totalColumns":2,"row":"multi_test_table,bbb,1529623399386.e04a2e293b4c1bbd21e87d569c11154e.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":1529623400821},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1529623400821}]},"ts":1529623400821} 2018-06-21 23:23:20,831 DEBUG [PEWorker-8] hbase.MetaTableAccessor(2153): Put {"totalColumns":2,"row":"multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":1529623400821},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1529623400821}]},"ts":1529623400821} 2018-06-21 23:23:20,831 DEBUG [PEWorker-8] hbase.MetaTableAccessor(2153): Put {"totalColumns":2,"row":"multi_test_table,uuu,1529623399386.40503e5e9a757a4ccf579bf29b8d78c6.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":1529623400821},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1529623400821}]},"ts":1529623400821} 2018-06-21 23:23:20,906 INFO [PEWorker-8] hbase.MetaTableAccessor(1528): Added 26 regions to meta. 2018-06-21 23:23:20,961 DEBUG [PEWorker-8] hbase.MetaTableAccessor(2153): Put {"totalColumns":1,"row":"multi_test_table","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1529623400961}]},"ts":1529623400961} 2018-06-21 23:23:20,968 INFO [PEWorker-8] hbase.MetaTableAccessor(1673): Updated tableName=multi_test_table, state=ENABLING in hbase:meta 2018-06-21 23:23:20,994 DEBUG [PEWorker-8] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region multi_test_table,jjj,1529623399386.573e96916daa12dfc083e39a253dc3a2. 2018-06-21 23:23:21,007 DEBUG [PEWorker-8] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region multi_test_table,hhh,1529623399386.91ab16f39c84b448bdc2e5a1e1fc801b. 2018-06-21 23:23:21,008 DEBUG [PEWorker-8] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region multi_test_table,ccc,1529623399386.124da0ae361b264a91ea989bcd12b459. 2018-06-21 23:23:21,011 DEBUG [PEWorker-8] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region multi_test_table,iii,1529623399386.04d71925444c7c823938ba3d9ad0504d. 2018-06-21 23:23:21,013 DEBUG [PEWorker-8] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region multi_test_table,fff,1529623399386.da9d0eefced1af8dfad2324a5d9e1bf7. 2018-06-21 23:23:21,016 DEBUG [PEWorker-8] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region multi_test_table,,1529623399386.cb0df54f72fae3cd39a03514dbbaa19c. 2018-06-21 23:23:21,018 DEBUG [PEWorker-8] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region multi_test_table,mmm,1529623399386.28b5f974b61112896f4c4f1127dc2b99. 2018-06-21 23:23:21,020 DEBUG [PEWorker-8] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region multi_test_table,ppp,1529623399386.36d0a953419e9f81636aebf73e7ed5df. 2018-06-21 23:23:21,022 DEBUG [PEWorker-8] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region multi_test_table,ddd,1529623399386.e66f718b26d1c4420497ccfb42f08321. 2018-06-21 23:23:21,024 DEBUG [PEWorker-8] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173. 2018-06-21 23:23:21,026 DEBUG [PEWorker-8] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region multi_test_table,ooo,1529623399386.1449fc3d526b1c6ab2fd158edec23277. 2018-06-21 23:23:21,027 DEBUG [PEWorker-8] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region multi_test_table,qqq,1529623399386.7480a111a1f600aa34b97a77e47640bc. 2018-06-21 23:23:21,028 DEBUG [PEWorker-8] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c. 2018-06-21 23:23:21,029 DEBUG [PEWorker-8] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region multi_test_table,sss,1529623399386.d4ac3b44d02d68507d6317e201672011. 2018-06-21 23:23:21,030 DEBUG [PEWorker-8] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region multi_test_table,ttt,1529623399386.dff6ab063ea78552b5e3c3ae7760ca5a. 2018-06-21 23:23:21,031 DEBUG [PEWorker-8] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region multi_test_table,xxx,1529623399386.77a32d153e5d8148ca9d599c6a3cb695. 2018-06-21 23:23:21,033 DEBUG [PEWorker-8] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region multi_test_table,www,1529623399386.a657e5e90b9697e4cf81bc4bbde44800. 2018-06-21 23:23:21,034 DEBUG [PEWorker-8] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region multi_test_table,yyy,1529623399386.7006d69958189f4e3b610c07f795d5d5. 2018-06-21 23:23:21,035 DEBUG [PEWorker-8] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region multi_test_table,zzz,1529623399386.794028a2b8fc0a516890763306629d4c. 2018-06-21 23:23:21,036 DEBUG [PEWorker-8] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region multi_test_table,eee,1529623399386.c793c2c3948aad84865d30ebf71a529e. 2018-06-21 23:23:21,037 DEBUG [PEWorker-8] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region multi_test_table,lll,1529623399386.77885b472341ce945c76aa06a4627146. 2018-06-21 23:23:21,039 DEBUG [PEWorker-8] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region multi_test_table,nnn,1529623399386.b1a8d69ffdd43e80f30a919effdc7c4e. 2018-06-21 23:23:21,040 DEBUG [PEWorker-8] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region multi_test_table,ggg,1529623399386.10cdebbb720d98123807841a0febf405. 2018-06-21 23:23:21,041 DEBUG [PEWorker-8] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region multi_test_table,bbb,1529623399386.e04a2e293b4c1bbd21e87d569c11154e. 2018-06-21 23:23:21,042 DEBUG [PEWorker-8] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95. 2018-06-21 23:23:21,043 DEBUG [PEWorker-8] balancer.RegionLocationFinder(280): HDFSBlocksDistribution not found in cache for region multi_test_table,uuu,1529623399386.40503e5e9a757a4ccf579bf29b8d78c6. 2018-06-21 23:23:21,051 INFO [PEWorker-8] procedure2.ProcedureExecutor(1516): Initialized subprocedures=[{pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=cb0df54f72fae3cd39a03514dbbaa19c, target=asf911.gq1.ygridcore.net,58235,1529623390382}, {pid=9, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=e04a2e293b4c1bbd21e87d569c11154e, target=asf911.gq1.ygridcore.net,58235,1529623390382}, {pid=10, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=124da0ae361b264a91ea989bcd12b459, target=asf911.gq1.ygridcore.net,52507,1529623390228}, {pid=11, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=e66f718b26d1c4420497ccfb42f08321, target=asf911.gq1.ygridcore.net,52507,1529623390228}, {pid=12, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=c793c2c3948aad84865d30ebf71a529e, target=asf911.gq1.ygridcore.net,40767,1529623390148}, {pid=13, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=da9d0eefced1af8dfad2324a5d9e1bf7, target=asf911.gq1.ygridcore.net,56613,1529623390297}, {pid=14, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=10cdebbb720d98123807841a0febf405, target=asf911.gq1.ygridcore.net,56613,1529623390297}, {pid=15, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=91ab16f39c84b448bdc2e5a1e1fc801b, target=asf911.gq1.ygridcore.net,40767,1529623390148}, {pid=16, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=04d71925444c7c823938ba3d9ad0504d, target=asf911.gq1.ygridcore.net,55146,1529623388534}, {pid=17, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=573e96916daa12dfc083e39a253dc3a2, target=asf911.gq1.ygridcore.net,59970,1529623390066}, {pid=18, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=fee67ae5d4f2b2be35fa516877a8d173, target=asf911.gq1.ygridcore.net,55146,1529623388534}, {pid=19, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=77885b472341ce945c76aa06a4627146, target=asf911.gq1.ygridcore.net,52507,1529623390228}, {pid=20, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=28b5f974b61112896f4c4f1127dc2b99, target=asf911.gq1.ygridcore.net,59970,1529623390066}, {pid=21, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=b1a8d69ffdd43e80f30a919effdc7c4e, target=asf911.gq1.ygridcore.net,55146,1529623388534}, {pid=22, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=1449fc3d526b1c6ab2fd158edec23277, target=asf911.gq1.ygridcore.net,56613,1529623390297}, {pid=23, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=36d0a953419e9f81636aebf73e7ed5df, target=asf911.gq1.ygridcore.net,40767,1529623390148}, {pid=24, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=7480a111a1f600aa34b97a77e47640bc, target=asf911.gq1.ygridcore.net,58235,1529623390382}, {pid=25, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=8e2537011449fca6295a357cbde76e6c, target=asf911.gq1.ygridcore.net,59970,1529623390066}, {pid=26, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=d4ac3b44d02d68507d6317e201672011, target=asf911.gq1.ygridcore.net,40767,1529623390148}, {pid=27, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=dff6ab063ea78552b5e3c3ae7760ca5a, target=asf911.gq1.ygridcore.net,52507,1529623390228}, {pid=28, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=40503e5e9a757a4ccf579bf29b8d78c6, target=asf911.gq1.ygridcore.net,40767,1529623390148}, {pid=29, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=0dfb3ebb8c759d632ec793ac0387ec95, target=asf911.gq1.ygridcore.net,59970,1529623390066}, {pid=30, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=a657e5e90b9697e4cf81bc4bbde44800, target=asf911.gq1.ygridcore.net,56613,1529623390297}, {pid=31, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=77a32d153e5d8148ca9d599c6a3cb695, target=asf911.gq1.ygridcore.net,55146,1529623388534}, {pid=32, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=7006d69958189f4e3b610c07f795d5d5, target=asf911.gq1.ygridcore.net,58235,1529623390382}, {pid=33, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=794028a2b8fc0a516890763306629d4c, target=asf911.gq1.ygridcore.net,59970,1529623390066}] 2018-06-21 23:23:21,168 INFO [PEWorker-9] procedure.MasterProcedureScheduler(696): pid=21, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=b1a8d69ffdd43e80f30a919effdc7c4e, target=asf911.gq1.ygridcore.net,55146,1529623388534 checking lock on b1a8d69ffdd43e80f30a919effdc7c4e 2018-06-21 23:23:21,169 INFO [PEWorker-11] procedure.MasterProcedureScheduler(696): pid=16, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=04d71925444c7c823938ba3d9ad0504d, target=asf911.gq1.ygridcore.net,55146,1529623388534 checking lock on 04d71925444c7c823938ba3d9ad0504d 2018-06-21 23:23:21,169 INFO [PEWorker-13] procedure.MasterProcedureScheduler(696): pid=15, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=91ab16f39c84b448bdc2e5a1e1fc801b, target=asf911.gq1.ygridcore.net,40767,1529623390148 checking lock on 91ab16f39c84b448bdc2e5a1e1fc801b 2018-06-21 23:23:21,170 INFO [PEWorker-2] procedure.MasterProcedureScheduler(696): pid=13, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=da9d0eefced1af8dfad2324a5d9e1bf7, target=asf911.gq1.ygridcore.net,56613,1529623390297 checking lock on da9d0eefced1af8dfad2324a5d9e1bf7 2018-06-21 23:23:21,170 INFO [PEWorker-3] procedure.MasterProcedureScheduler(696): pid=11, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=e66f718b26d1c4420497ccfb42f08321, target=asf911.gq1.ygridcore.net,52507,1529623390228 checking lock on e66f718b26d1c4420497ccfb42f08321 2018-06-21 23:23:21,170 INFO [PEWorker-5] procedure.MasterProcedureScheduler(696): pid=9, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=e04a2e293b4c1bbd21e87d569c11154e, target=asf911.gq1.ygridcore.net,58235,1529623390382 checking lock on e04a2e293b4c1bbd21e87d569c11154e 2018-06-21 23:23:21,171 INFO [PEWorker-16] procedure.MasterProcedureScheduler(696): pid=20, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=28b5f974b61112896f4c4f1127dc2b99, target=asf911.gq1.ygridcore.net,59970,1529623390066 checking lock on 28b5f974b61112896f4c4f1127dc2b99 2018-06-21 23:23:21,174 INFO [PEWorker-1] procedure.MasterProcedureScheduler(696): pid=19, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=77885b472341ce945c76aa06a4627146, target=asf911.gq1.ygridcore.net,52507,1529623390228 checking lock on 77885b472341ce945c76aa06a4627146 2018-06-21 23:23:21,174 INFO [PEWorker-10] procedure.MasterProcedureScheduler(696): pid=18, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=fee67ae5d4f2b2be35fa516877a8d173, target=asf911.gq1.ygridcore.net,55146,1529623388534 checking lock on fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:21,175 INFO [PEWorker-12] procedure.MasterProcedureScheduler(696): pid=17, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=573e96916daa12dfc083e39a253dc3a2, target=asf911.gq1.ygridcore.net,59970,1529623390066 checking lock on 573e96916daa12dfc083e39a253dc3a2 2018-06-21 23:23:21,175 INFO [PEWorker-14] procedure.MasterProcedureScheduler(696): pid=14, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=10cdebbb720d98123807841a0febf405, target=asf911.gq1.ygridcore.net,56613,1529623390297 checking lock on 10cdebbb720d98123807841a0febf405 2018-06-21 23:23:21,175 INFO [PEWorker-15] procedure.MasterProcedureScheduler(696): pid=12, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=c793c2c3948aad84865d30ebf71a529e, target=asf911.gq1.ygridcore.net,40767,1529623390148 checking lock on c793c2c3948aad84865d30ebf71a529e 2018-06-21 23:23:21,176 INFO [PEWorker-9] assignment.AssignProcedure(218): Starting pid=21, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=b1a8d69ffdd43e80f30a919effdc7c4e, target=asf911.gq1.ygridcore.net,55146,1529623388534; rit=OFFLINE, location=asf911.gq1.ygridcore.net,55146,1529623388534; forceNewPlan=false, retain=false 2018-06-21 23:23:21,176 INFO [PEWorker-4] procedure.MasterProcedureScheduler(696): pid=10, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=124da0ae361b264a91ea989bcd12b459, target=asf911.gq1.ygridcore.net,52507,1529623390228 checking lock on 124da0ae361b264a91ea989bcd12b459 2018-06-21 23:23:21,176 INFO [PEWorker-6] procedure.MasterProcedureScheduler(696): pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=cb0df54f72fae3cd39a03514dbbaa19c, target=asf911.gq1.ygridcore.net,58235,1529623390382 checking lock on cb0df54f72fae3cd39a03514dbbaa19c 2018-06-21 23:23:21,176 INFO [PEWorker-7] procedure.MasterProcedureScheduler(696): pid=33, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=794028a2b8fc0a516890763306629d4c, target=asf911.gq1.ygridcore.net,59970,1529623390066 checking lock on 794028a2b8fc0a516890763306629d4c 2018-06-21 23:23:21,177 INFO [PEWorker-8] procedure.MasterProcedureScheduler(696): pid=32, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=7006d69958189f4e3b610c07f795d5d5, target=asf911.gq1.ygridcore.net,58235,1529623390382 checking lock on 7006d69958189f4e3b610c07f795d5d5 2018-06-21 23:23:21,177 INFO [PEWorker-2] assignment.AssignProcedure(218): Starting pid=13, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=da9d0eefced1af8dfad2324a5d9e1bf7, target=asf911.gq1.ygridcore.net,56613,1529623390297; rit=OFFLINE, location=asf911.gq1.ygridcore.net,56613,1529623390297; forceNewPlan=false, retain=false 2018-06-21 23:23:21,178 INFO [PEWorker-3] assignment.AssignProcedure(218): Starting pid=11, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=e66f718b26d1c4420497ccfb42f08321, target=asf911.gq1.ygridcore.net,52507,1529623390228; rit=OFFLINE, location=asf911.gq1.ygridcore.net,52507,1529623390228; forceNewPlan=false, retain=false 2018-06-21 23:23:21,178 INFO [PEWorker-13] assignment.AssignProcedure(218): Starting pid=15, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=91ab16f39c84b448bdc2e5a1e1fc801b, target=asf911.gq1.ygridcore.net,40767,1529623390148; rit=OFFLINE, location=asf911.gq1.ygridcore.net,40767,1529623390148; forceNewPlan=false, retain=false 2018-06-21 23:23:21,179 INFO [PEWorker-5] assignment.AssignProcedure(218): Starting pid=9, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=e04a2e293b4c1bbd21e87d569c11154e, target=asf911.gq1.ygridcore.net,58235,1529623390382; rit=OFFLINE, location=asf911.gq1.ygridcore.net,58235,1529623390382; forceNewPlan=false, retain=false 2018-06-21 23:23:21,179 INFO [PEWorker-16] assignment.AssignProcedure(218): Starting pid=20, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=28b5f974b61112896f4c4f1127dc2b99, target=asf911.gq1.ygridcore.net,59970,1529623390066; rit=OFFLINE, location=asf911.gq1.ygridcore.net,59970,1529623390066; forceNewPlan=false, retain=false 2018-06-21 23:23:21,179 INFO [PEWorker-1] assignment.AssignProcedure(218): Starting pid=19, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=77885b472341ce945c76aa06a4627146, target=asf911.gq1.ygridcore.net,52507,1529623390228; rit=OFFLINE, location=asf911.gq1.ygridcore.net,52507,1529623390228; forceNewPlan=false, retain=false 2018-06-21 23:23:21,179 INFO [PEWorker-10] assignment.AssignProcedure(218): Starting pid=18, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=fee67ae5d4f2b2be35fa516877a8d173, target=asf911.gq1.ygridcore.net,55146,1529623388534; rit=OFFLINE, location=asf911.gq1.ygridcore.net,55146,1529623388534; forceNewPlan=false, retain=false 2018-06-21 23:23:21,179 INFO [PEWorker-14] assignment.AssignProcedure(218): Starting pid=14, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=10cdebbb720d98123807841a0febf405, target=asf911.gq1.ygridcore.net,56613,1529623390297; rit=OFFLINE, location=asf911.gq1.ygridcore.net,56613,1529623390297; forceNewPlan=false, retain=false 2018-06-21 23:23:21,180 INFO [PEWorker-12] assignment.AssignProcedure(218): Starting pid=17, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=573e96916daa12dfc083e39a253dc3a2, target=asf911.gq1.ygridcore.net,59970,1529623390066; rit=OFFLINE, location=asf911.gq1.ygridcore.net,59970,1529623390066; forceNewPlan=false, retain=false 2018-06-21 23:23:21,180 INFO [PEWorker-11] assignment.AssignProcedure(218): Starting pid=16, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=04d71925444c7c823938ba3d9ad0504d, target=asf911.gq1.ygridcore.net,55146,1529623388534; rit=OFFLINE, location=asf911.gq1.ygridcore.net,55146,1529623388534; forceNewPlan=false, retain=false 2018-06-21 23:23:21,180 INFO [PEWorker-15] assignment.AssignProcedure(218): Starting pid=12, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=c793c2c3948aad84865d30ebf71a529e, target=asf911.gq1.ygridcore.net,40767,1529623390148; rit=OFFLINE, location=asf911.gq1.ygridcore.net,40767,1529623390148; forceNewPlan=false, retain=false 2018-06-21 23:23:21,180 INFO [PEWorker-4] assignment.AssignProcedure(218): Starting pid=10, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=124da0ae361b264a91ea989bcd12b459, target=asf911.gq1.ygridcore.net,52507,1529623390228; rit=OFFLINE, location=asf911.gq1.ygridcore.net,52507,1529623390228; forceNewPlan=false, retain=false 2018-06-21 23:23:21,181 INFO [PEWorker-6] assignment.AssignProcedure(218): Starting pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=cb0df54f72fae3cd39a03514dbbaa19c, target=asf911.gq1.ygridcore.net,58235,1529623390382; rit=OFFLINE, location=asf911.gq1.ygridcore.net,58235,1529623390382; forceNewPlan=false, retain=false 2018-06-21 23:23:21,181 INFO [PEWorker-7] assignment.AssignProcedure(218): Starting pid=33, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=794028a2b8fc0a516890763306629d4c, target=asf911.gq1.ygridcore.net,59970,1529623390066; rit=OFFLINE, location=asf911.gq1.ygridcore.net,59970,1529623390066; forceNewPlan=false, retain=false 2018-06-21 23:23:21,182 INFO [PEWorker-8] assignment.AssignProcedure(218): Starting pid=32, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=7006d69958189f4e3b610c07f795d5d5, target=asf911.gq1.ygridcore.net,58235,1529623390382; rit=OFFLINE, location=asf911.gq1.ygridcore.net,58235,1529623390382; forceNewPlan=false, retain=false 2018-06-21 23:23:21,307 INFO [PEWorker-9] procedure.MasterProcedureScheduler(696): pid=31, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=77a32d153e5d8148ca9d599c6a3cb695, target=asf911.gq1.ygridcore.net,55146,1529623388534 checking lock on 77a32d153e5d8148ca9d599c6a3cb695 2018-06-21 23:23:21,307 INFO [PEWorker-2] procedure.MasterProcedureScheduler(696): pid=30, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=a657e5e90b9697e4cf81bc4bbde44800, target=asf911.gq1.ygridcore.net,56613,1529623390297 checking lock on a657e5e90b9697e4cf81bc4bbde44800 2018-06-21 23:23:21,308 INFO [PEWorker-5] procedure.MasterProcedureScheduler(696): pid=26, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=d4ac3b44d02d68507d6317e201672011, target=asf911.gq1.ygridcore.net,40767,1529623390148 checking lock on d4ac3b44d02d68507d6317e201672011 2018-06-21 23:23:21,308 INFO [PEWorker-16] procedure.MasterProcedureScheduler(696): pid=25, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=8e2537011449fca6295a357cbde76e6c, target=asf911.gq1.ygridcore.net,59970,1529623390066 checking lock on 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:21,309 INFO [PEWorker-1] procedure.MasterProcedureScheduler(696): pid=24, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=7480a111a1f600aa34b97a77e47640bc, target=asf911.gq1.ygridcore.net,58235,1529623390382 checking lock on 7480a111a1f600aa34b97a77e47640bc 2018-06-21 23:23:21,310 INFO [PEWorker-14] procedure.MasterProcedureScheduler(696): pid=22, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=1449fc3d526b1c6ab2fd158edec23277, target=asf911.gq1.ygridcore.net,56613,1529623390297 checking lock on 1449fc3d526b1c6ab2fd158edec23277 2018-06-21 23:23:21,310 INFO [PEWorker-3] procedure.MasterProcedureScheduler(696): pid=29, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=0dfb3ebb8c759d632ec793ac0387ec95, target=asf911.gq1.ygridcore.net,59970,1529623390066 checking lock on 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:21,311 INFO [PEWorker-6] procedure.MasterProcedureScheduler(696): pid=28, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=40503e5e9a757a4ccf579bf29b8d78c6, target=asf911.gq1.ygridcore.net,40767,1529623390148 checking lock on 40503e5e9a757a4ccf579bf29b8d78c6 2018-06-21 23:23:21,311 INFO [PEWorker-13] procedure.MasterProcedureScheduler(696): pid=27, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=dff6ab063ea78552b5e3c3ae7760ca5a, target=asf911.gq1.ygridcore.net,52507,1529623390228 checking lock on dff6ab063ea78552b5e3c3ae7760ca5a 2018-06-21 23:23:21,311 INFO [PEWorker-2] assignment.AssignProcedure(218): Starting pid=30, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=a657e5e90b9697e4cf81bc4bbde44800, target=asf911.gq1.ygridcore.net,56613,1529623390297; rit=OFFLINE, location=asf911.gq1.ygridcore.net,56613,1529623390297; forceNewPlan=false, retain=false 2018-06-21 23:23:21,311 INFO [PEWorker-10] procedure.MasterProcedureScheduler(696): pid=23, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=36d0a953419e9f81636aebf73e7ed5df, target=asf911.gq1.ygridcore.net,40767,1529623390148 checking lock on 36d0a953419e9f81636aebf73e7ed5df 2018-06-21 23:23:21,311 INFO [PEWorker-5] assignment.AssignProcedure(218): Starting pid=26, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=d4ac3b44d02d68507d6317e201672011, target=asf911.gq1.ygridcore.net,40767,1529623390148; rit=OFFLINE, location=asf911.gq1.ygridcore.net,40767,1529623390148; forceNewPlan=false, retain=false 2018-06-21 23:23:21,311 INFO [PEWorker-16] assignment.AssignProcedure(218): Starting pid=25, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=8e2537011449fca6295a357cbde76e6c, target=asf911.gq1.ygridcore.net,59970,1529623390066; rit=OFFLINE, location=asf911.gq1.ygridcore.net,59970,1529623390066; forceNewPlan=false, retain=false 2018-06-21 23:23:21,311 INFO [PEWorker-9] assignment.AssignProcedure(218): Starting pid=31, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=77a32d153e5d8148ca9d599c6a3cb695, target=asf911.gq1.ygridcore.net,55146,1529623388534; rit=OFFLINE, location=asf911.gq1.ygridcore.net,55146,1529623388534; forceNewPlan=false, retain=false 2018-06-21 23:23:21,313 INFO [PEWorker-1] assignment.AssignProcedure(218): Starting pid=24, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=7480a111a1f600aa34b97a77e47640bc, target=asf911.gq1.ygridcore.net,58235,1529623390382; rit=OFFLINE, location=asf911.gq1.ygridcore.net,58235,1529623390382; forceNewPlan=false, retain=false 2018-06-21 23:23:21,313 INFO [PEWorker-14] assignment.AssignProcedure(218): Starting pid=22, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=1449fc3d526b1c6ab2fd158edec23277, target=asf911.gq1.ygridcore.net,56613,1529623390297; rit=OFFLINE, location=asf911.gq1.ygridcore.net,56613,1529623390297; forceNewPlan=false, retain=false 2018-06-21 23:23:21,314 INFO [PEWorker-3] assignment.AssignProcedure(218): Starting pid=29, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=0dfb3ebb8c759d632ec793ac0387ec95, target=asf911.gq1.ygridcore.net,59970,1529623390066; rit=OFFLINE, location=asf911.gq1.ygridcore.net,59970,1529623390066; forceNewPlan=false, retain=false 2018-06-21 23:23:21,314 INFO [PEWorker-6] assignment.AssignProcedure(218): Starting pid=28, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=40503e5e9a757a4ccf579bf29b8d78c6, target=asf911.gq1.ygridcore.net,40767,1529623390148; rit=OFFLINE, location=asf911.gq1.ygridcore.net,40767,1529623390148; forceNewPlan=false, retain=false 2018-06-21 23:23:21,314 INFO [PEWorker-13] assignment.AssignProcedure(218): Starting pid=27, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=dff6ab063ea78552b5e3c3ae7760ca5a, target=asf911.gq1.ygridcore.net,52507,1529623390228; rit=OFFLINE, location=asf911.gq1.ygridcore.net,52507,1529623390228; forceNewPlan=false, retain=false 2018-06-21 23:23:21,314 INFO [PEWorker-10] assignment.AssignProcedure(218): Starting pid=23, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=36d0a953419e9f81636aebf73e7ed5df, target=asf911.gq1.ygridcore.net,40767,1529623390148; rit=OFFLINE, location=asf911.gq1.ygridcore.net,40767,1529623390148; forceNewPlan=false, retain=false 2018-06-21 23:23:21,326 INFO [master/asf911:0] balancer.BaseLoadBalancer(1497): Reassigned 26 regions. 26 retained the pre-restart assignment. 2018-06-21 23:23:21,349 INFO [PEWorker-12] assignment.RegionStateStore(198): pid=17 updating hbase:meta row=573e96916daa12dfc083e39a253dc3a2, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,350 INFO [PEWorker-15] assignment.RegionStateStore(198): pid=25 updating hbase:meta row=8e2537011449fca6295a357cbde76e6c, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,350 INFO [PEWorker-4] assignment.RegionStateStore(198): pid=33 updating hbase:meta row=794028a2b8fc0a516890763306629d4c, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,350 INFO [PEWorker-11] assignment.RegionStateStore(198): pid=20 updating hbase:meta row=28b5f974b61112896f4c4f1127dc2b99, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,350 INFO [PEWorker-7] assignment.RegionStateStore(198): pid=29 updating hbase:meta row=0dfb3ebb8c759d632ec793ac0387ec95, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,360 INFO [PEWorker-12] assignment.RegionTransitionProcedure(241): Dispatch pid=17, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=573e96916daa12dfc083e39a253dc3a2, target=asf911.gq1.ygridcore.net,59970,1529623390066; rit=OPENING, location=asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,361 INFO [PEWorker-15] assignment.RegionTransitionProcedure(241): Dispatch pid=25, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=8e2537011449fca6295a357cbde76e6c, target=asf911.gq1.ygridcore.net,59970,1529623390066; rit=OPENING, location=asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,368 INFO [PEWorker-4] assignment.RegionTransitionProcedure(241): Dispatch pid=33, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=794028a2b8fc0a516890763306629d4c, target=asf911.gq1.ygridcore.net,59970,1529623390066; rit=OPENING, location=asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,368 INFO [PEWorker-7] assignment.RegionTransitionProcedure(241): Dispatch pid=29, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=0dfb3ebb8c759d632ec793ac0387ec95, target=asf911.gq1.ygridcore.net,59970,1529623390066; rit=OPENING, location=asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,368 INFO [PEWorker-11] assignment.RegionTransitionProcedure(241): Dispatch pid=20, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=28b5f974b61112896f4c4f1127dc2b99, target=asf911.gq1.ygridcore.net,59970,1529623390066; rit=OPENING, location=asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,378 INFO [PEWorker-5] assignment.RegionStateStore(198): pid=8 updating hbase:meta row=cb0df54f72fae3cd39a03514dbbaa19c, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:21,378 INFO [PEWorker-16] assignment.RegionStateStore(198): pid=32 updating hbase:meta row=7006d69958189f4e3b610c07f795d5d5, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:21,378 INFO [PEWorker-2] assignment.RegionStateStore(198): pid=24 updating hbase:meta row=7480a111a1f600aa34b97a77e47640bc, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:21,378 INFO [PEWorker-8] assignment.RegionStateStore(198): pid=9 updating hbase:meta row=e04a2e293b4c1bbd21e87d569c11154e, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:21,407 INFO [PEWorker-5] assignment.RegionTransitionProcedure(241): Dispatch pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=cb0df54f72fae3cd39a03514dbbaa19c, target=asf911.gq1.ygridcore.net,58235,1529623390382; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:21,411 INFO [PEWorker-16] assignment.RegionTransitionProcedure(241): Dispatch pid=32, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=7006d69958189f4e3b610c07f795d5d5, target=asf911.gq1.ygridcore.net,58235,1529623390382; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:21,411 INFO [PEWorker-2] assignment.RegionTransitionProcedure(241): Dispatch pid=24, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=7480a111a1f600aa34b97a77e47640bc, target=asf911.gq1.ygridcore.net,58235,1529623390382; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:21,414 INFO [PEWorker-8] assignment.RegionTransitionProcedure(241): Dispatch pid=9, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=e04a2e293b4c1bbd21e87d569c11154e, target=asf911.gq1.ygridcore.net,58235,1529623390382; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:21,513 DEBUG [RSProcedureDispatcher-pool3-t3] master.ServerManager(746): New admin connection to asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,518 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-06-21 23:23:21,518 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 51512 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:21,519 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=59970] regionserver.RSRpcServices(1980): Open multi_test_table,zzz,1529623399386.794028a2b8fc0a516890763306629d4c. 2018-06-21 23:23:21,530 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=59970] regionserver.RSRpcServices(1980): Open multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c. 2018-06-21 23:23:21,531 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7094): Opening region: {ENCODED => 794028a2b8fc0a516890763306629d4c, NAME => 'multi_test_table,zzz,1529623399386.794028a2b8fc0a516890763306629d4c.', STARTKEY => 'zzz', ENDKEY => ''} 2018-06-21 23:23:21,532 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=59970] regionserver.RSRpcServices(1980): Open multi_test_table,mmm,1529623399386.28b5f974b61112896f4c4f1127dc2b99. 2018-06-21 23:23:21,533 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7094): Opening region: {ENCODED => 8e2537011449fca6295a357cbde76e6c, NAME => 'multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c.', STARTKEY => 'rrr', ENDKEY => 'sss'} 2018-06-21 23:23:21,533 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table 794028a2b8fc0a516890763306629d4c 2018-06-21 23:23:21,533 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=59970] regionserver.RSRpcServices(1980): Open multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95. 2018-06-21 23:23:21,534 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(827): Instantiated multi_test_table,zzz,1529623399386.794028a2b8fc0a516890763306629d4c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:21,535 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=59970] regionserver.RSRpcServices(1980): Open multi_test_table,jjj,1529623399386.573e96916daa12dfc083e39a253dc3a2. 2018-06-21 23:23:21,535 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7094): Opening region: {ENCODED => 28b5f974b61112896f4c4f1127dc2b99, NAME => 'multi_test_table,mmm,1529623399386.28b5f974b61112896f4c4f1127dc2b99.', STARTKEY => 'mmm', ENDKEY => 'nnn'} 2018-06-21 23:23:21,535 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7134): checking encryption for 794028a2b8fc0a516890763306629d4c 2018-06-21 23:23:21,535 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table 28b5f974b61112896f4c4f1127dc2b99 2018-06-21 23:23:21,535 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:21,535 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(827): Instantiated multi_test_table,mmm,1529623399386.28b5f974b61112896f4c4f1127dc2b99.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:21,535 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7139): checking classloading for 794028a2b8fc0a516890763306629d4c 2018-06-21 23:23:21,536 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7134): checking encryption for 28b5f974b61112896f4c4f1127dc2b99 2018-06-21 23:23:21,536 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7139): checking classloading for 28b5f974b61112896f4c4f1127dc2b99 2018-06-21 23:23:21,536 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(827): Instantiated multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:21,538 INFO [PEWorker-16] assignment.RegionStateStore(198): pid=31 updating hbase:meta row=77a32d153e5d8148ca9d599c6a3cb695, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:21,538 INFO [PEWorker-8] assignment.RegionStateStore(198): pid=18 updating hbase:meta row=fee67ae5d4f2b2be35fa516877a8d173, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:21,538 INFO [PEWorker-5] assignment.RegionStateStore(198): pid=21 updating hbase:meta row=b1a8d69ffdd43e80f30a919effdc7c4e, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:21,536 INFO [PEWorker-3] assignment.RegionStateStore(198): pid=13 updating hbase:meta row=da9d0eefced1af8dfad2324a5d9e1bf7, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:21,536 INFO [PEWorker-9] assignment.RegionStateStore(198): pid=14 updating hbase:meta row=10cdebbb720d98123807841a0febf405, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:21,536 INFO [PEWorker-1] assignment.RegionStateStore(198): pid=30 updating hbase:meta row=a657e5e90b9697e4cf81bc4bbde44800, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:21,536 INFO [PEWorker-14] assignment.RegionStateStore(198): pid=22 updating hbase:meta row=1449fc3d526b1c6ab2fd158edec23277, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:21,538 INFO [PEWorker-2] assignment.RegionStateStore(198): pid=16 updating hbase:meta row=04d71925444c7c823938ba3d9ad0504d, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:21,538 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7134): checking encryption for 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:21,543 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7139): checking classloading for 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:21,555 INFO [PEWorker-16] assignment.RegionTransitionProcedure(241): Dispatch pid=31, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=77a32d153e5d8148ca9d599c6a3cb695, target=asf911.gq1.ygridcore.net,55146,1529623388534; rit=OPENING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:21,557 INFO [PEWorker-3] assignment.RegionTransitionProcedure(241): Dispatch pid=13, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=da9d0eefced1af8dfad2324a5d9e1bf7, target=asf911.gq1.ygridcore.net,56613,1529623390297; rit=OPENING, location=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:21,557 INFO [PEWorker-5] assignment.RegionTransitionProcedure(241): Dispatch pid=21, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=b1a8d69ffdd43e80f30a919effdc7c4e, target=asf911.gq1.ygridcore.net,55146,1529623388534; rit=OPENING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:21,557 INFO [PEWorker-2] assignment.RegionTransitionProcedure(241): Dispatch pid=16, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=04d71925444c7c823938ba3d9ad0504d, target=asf911.gq1.ygridcore.net,55146,1529623388534; rit=OPENING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:21,559 INFO [PEWorker-8] assignment.RegionTransitionProcedure(241): Dispatch pid=18, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=fee67ae5d4f2b2be35fa516877a8d173, target=asf911.gq1.ygridcore.net,55146,1529623388534; rit=OPENING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:21,559 INFO [PEWorker-1] assignment.RegionTransitionProcedure(241): Dispatch pid=30, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=a657e5e90b9697e4cf81bc4bbde44800, target=asf911.gq1.ygridcore.net,56613,1529623390297; rit=OPENING, location=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:21,564 DEBUG [StoreOpener-28b5f974b61112896f4c4f1127dc2b99-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/28b5f974b61112896f4c4f1127dc2b99/family 2018-06-21 23:23:21,567 DEBUG [StoreOpener-28b5f974b61112896f4c4f1127dc2b99-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/28b5f974b61112896f4c4f1127dc2b99/family 2018-06-21 23:23:21,567 DEBUG [StoreOpener-794028a2b8fc0a516890763306629d4c-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/794028a2b8fc0a516890763306629d4c/family 2018-06-21 23:23:21,567 DEBUG [StoreOpener-794028a2b8fc0a516890763306629d4c-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/794028a2b8fc0a516890763306629d4c/family 2018-06-21 23:23:21,566 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58235] regionserver.RSRpcServices(1980): Open multi_test_table,yyy,1529623399386.7006d69958189f4e3b610c07f795d5d5. 2018-06-21 23:23:21,565 INFO [PEWorker-14] assignment.RegionTransitionProcedure(241): Dispatch pid=22, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=1449fc3d526b1c6ab2fd158edec23277, target=asf911.gq1.ygridcore.net,56613,1529623390297; rit=OPENING, location=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:21,565 INFO [PEWorker-9] assignment.RegionTransitionProcedure(241): Dispatch pid=14, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=10cdebbb720d98123807841a0febf405, target=asf911.gq1.ygridcore.net,56613,1529623390297; rit=OPENING, location=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:21,567 DEBUG [StoreOpener-8e2537011449fca6295a357cbde76e6c-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/family 2018-06-21 23:23:21,569 DEBUG [StoreOpener-8e2537011449fca6295a357cbde76e6c-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/family 2018-06-21 23:23:21,578 INFO [StoreOpener-794028a2b8fc0a516890763306629d4c-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:21,579 INFO [StoreOpener-8e2537011449fca6295a357cbde76e6c-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:21,579 INFO [StoreOpener-794028a2b8fc0a516890763306629d4c-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:21,579 INFO [StoreOpener-28b5f974b61112896f4c4f1127dc2b99-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:21,579 INFO [StoreOpener-8e2537011449fca6295a357cbde76e6c-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:21,580 INFO [StoreOpener-28b5f974b61112896f4c4f1127dc2b99-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:21,581 INFO [StoreOpener-794028a2b8fc0a516890763306629d4c-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:21,581 INFO [StoreOpener-28b5f974b61112896f4c4f1127dc2b99-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:21,581 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(923): replaying wal for 794028a2b8fc0a516890763306629d4c 2018-06-21 23:23:21,583 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(923): replaying wal for 28b5f974b61112896f4c4f1127dc2b99 2018-06-21 23:23:21,588 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7094): Opening region: {ENCODED => 7006d69958189f4e3b610c07f795d5d5, NAME => 'multi_test_table,yyy,1529623399386.7006d69958189f4e3b610c07f795d5d5.', STARTKEY => 'yyy', ENDKEY => 'zzz'} 2018-06-21 23:23:21,588 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58235] regionserver.RSRpcServices(1980): Open multi_test_table,qqq,1529623399386.7480a111a1f600aa34b97a77e47640bc. 2018-06-21 23:23:21,588 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/794028a2b8fc0a516890763306629d4c 2018-06-21 23:23:21,592 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(931): stopping wal replay for 794028a2b8fc0a516890763306629d4c 2018-06-21 23:23:21,592 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(943): Cleaning up temporary data for 794028a2b8fc0a516890763306629d4c 2018-06-21 23:23:21,589 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58235] regionserver.RSRpcServices(1980): Open multi_test_table,bbb,1529623399386.e04a2e293b4c1bbd21e87d569c11154e. 2018-06-21 23:23:21,589 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table 7006d69958189f4e3b610c07f795d5d5 2018-06-21 23:23:21,593 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(954): Cleaning up detritus for 794028a2b8fc0a516890763306629d4c 2018-06-21 23:23:21,589 INFO [StoreOpener-8e2537011449fca6295a357cbde76e6c-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:21,589 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/28b5f974b61112896f4c4f1127dc2b99 2018-06-21 23:23:21,595 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(931): stopping wal replay for 28b5f974b61112896f4c4f1127dc2b99 2018-06-21 23:23:21,595 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(943): Cleaning up temporary data for 28b5f974b61112896f4c4f1127dc2b99 2018-06-21 23:23:21,595 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7094): Opening region: {ENCODED => e04a2e293b4c1bbd21e87d569c11154e, NAME => 'multi_test_table,bbb,1529623399386.e04a2e293b4c1bbd21e87d569c11154e.', STARTKEY => 'bbb', ENDKEY => 'ccc'} 2018-06-21 23:23:21,593 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(827): Instantiated multi_test_table,yyy,1529623399386.7006d69958189f4e3b610c07f795d5d5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:21,593 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58235] regionserver.RSRpcServices(1980): Open multi_test_table,,1529623399386.cb0df54f72fae3cd39a03514dbbaa19c. 2018-06-21 23:23:21,599 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table e04a2e293b4c1bbd21e87d569c11154e 2018-06-21 23:23:21,600 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(827): Instantiated multi_test_table,bbb,1529623399386.e04a2e293b4c1bbd21e87d569c11154e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:21,600 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7134): checking encryption for e04a2e293b4c1bbd21e87d569c11154e 2018-06-21 23:23:21,600 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7139): checking classloading for e04a2e293b4c1bbd21e87d569c11154e 2018-06-21 23:23:21,592 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7094): Opening region: {ENCODED => 7480a111a1f600aa34b97a77e47640bc, NAME => 'multi_test_table,qqq,1529623399386.7480a111a1f600aa34b97a77e47640bc.', STARTKEY => 'qqq', ENDKEY => 'rrr'} 2018-06-21 23:23:21,599 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7134): checking encryption for 7006d69958189f4e3b610c07f795d5d5 2018-06-21 23:23:21,599 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(954): Cleaning up detritus for 28b5f974b61112896f4c4f1127dc2b99 2018-06-21 23:23:21,595 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(923): replaying wal for 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:21,601 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7139): checking classloading for 7006d69958189f4e3b610c07f795d5d5 2018-06-21 23:23:21,600 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table 7480a111a1f600aa34b97a77e47640bc 2018-06-21 23:23:21,602 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(827): Instantiated multi_test_table,qqq,1529623399386.7480a111a1f600aa34b97a77e47640bc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:21,602 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:21,608 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(931): stopping wal replay for 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:21,608 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(943): Cleaning up temporary data for 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:21,609 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(976): writing seq id for 28b5f974b61112896f4c4f1127dc2b99 2018-06-21 23:23:21,609 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(954): Cleaning up detritus for 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:21,608 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(976): writing seq id for 794028a2b8fc0a516890763306629d4c 2018-06-21 23:23:21,603 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7134): checking encryption for 7480a111a1f600aa34b97a77e47640bc 2018-06-21 23:23:21,612 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7139): checking classloading for 7480a111a1f600aa34b97a77e47640bc 2018-06-21 23:23:21,612 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(976): writing seq id for 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:21,619 DEBUG [StoreOpener-e04a2e293b4c1bbd21e87d569c11154e-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/e04a2e293b4c1bbd21e87d569c11154e/family 2018-06-21 23:23:21,620 DEBUG [StoreOpener-e04a2e293b4c1bbd21e87d569c11154e-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/e04a2e293b4c1bbd21e87d569c11154e/family 2018-06-21 23:23:21,621 INFO [StoreOpener-e04a2e293b4c1bbd21e87d569c11154e-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:21,621 INFO [StoreOpener-e04a2e293b4c1bbd21e87d569c11154e-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:21,621 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-06-21 23:23:21,621 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(980): Opened 8e2537011449fca6295a357cbde76e6c; next sequenceid=2 2018-06-21 23:23:21,622 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(987): Running coprocessor post-open hooks for 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:21,622 DEBUG [StoreOpener-7006d69958189f4e3b610c07f795d5d5-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/7006d69958189f4e3b610c07f795d5d5/family 2018-06-21 23:23:21,622 DEBUG [StoreOpener-7006d69958189f4e3b610c07f795d5d5-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/7006d69958189f4e3b610c07f795d5d5/family 2018-06-21 23:23:21,628 INFO [StoreOpener-e04a2e293b4c1bbd21e87d569c11154e-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:21,628 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/28b5f974b61112896f4c4f1127dc2b99/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-06-21 23:23:21,628 INFO [StoreOpener-7006d69958189f4e3b610c07f795d5d5-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:21,628 INFO [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(980): Opened 28b5f974b61112896f4c4f1127dc2b99; next sequenceid=2 2018-06-21 23:23:21,629 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(987): Running coprocessor post-open hooks for 28b5f974b61112896f4c4f1127dc2b99 2018-06-21 23:23:21,629 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(923): replaying wal for e04a2e293b4c1bbd21e87d569c11154e 2018-06-21 23:23:21,629 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/794028a2b8fc0a516890763306629d4c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-06-21 23:23:21,632 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(980): Opened 794028a2b8fc0a516890763306629d4c; next sequenceid=2 2018-06-21 23:23:21,629 INFO [StoreOpener-7006d69958189f4e3b610c07f795d5d5-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:21,632 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(987): Running coprocessor post-open hooks for 794028a2b8fc0a516890763306629d4c 2018-06-21 23:23:21,633 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/e04a2e293b4c1bbd21e87d569c11154e 2018-06-21 23:23:21,633 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(931): stopping wal replay for e04a2e293b4c1bbd21e87d569c11154e 2018-06-21 23:23:21,633 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(943): Cleaning up temporary data for e04a2e293b4c1bbd21e87d569c11154e 2018-06-21 23:23:21,633 INFO [StoreOpener-7006d69958189f4e3b610c07f795d5d5-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:21,634 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(923): replaying wal for 7006d69958189f4e3b610c07f795d5d5 2018-06-21 23:23:21,635 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(954): Cleaning up detritus for e04a2e293b4c1bbd21e87d569c11154e 2018-06-21 23:23:21,636 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/7006d69958189f4e3b610c07f795d5d5 2018-06-21 23:23:21,636 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(931): stopping wal replay for 7006d69958189f4e3b610c07f795d5d5 2018-06-21 23:23:21,636 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(943): Cleaning up temporary data for 7006d69958189f4e3b610c07f795d5d5 2018-06-21 23:23:21,636 DEBUG [StoreOpener-7480a111a1f600aa34b97a77e47640bc-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/7480a111a1f600aa34b97a77e47640bc/family 2018-06-21 23:23:21,636 DEBUG [StoreOpener-7480a111a1f600aa34b97a77e47640bc-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/7480a111a1f600aa34b97a77e47640bc/family 2018-06-21 23:23:21,637 INFO [StoreOpener-7480a111a1f600aa34b97a77e47640bc-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:21,638 INFO [StoreOpener-7480a111a1f600aa34b97a77e47640bc-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:21,640 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(954): Cleaning up detritus for 7006d69958189f4e3b610c07f795d5d5 2018-06-21 23:23:21,640 INFO [PostOpenDeployTasks:8e2537011449fca6295a357cbde76e6c] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c. 2018-06-21 23:23:21,640 INFO [PostOpenDeployTasks:794028a2b8fc0a516890763306629d4c] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,zzz,1529623399386.794028a2b8fc0a516890763306629d4c. 2018-06-21 23:23:21,641 INFO [PostOpenDeployTasks:28b5f974b61112896f4c4f1127dc2b99] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,mmm,1529623399386.28b5f974b61112896f4c4f1127dc2b99. 2018-06-21 23:23:21,644 INFO [StoreOpener-7480a111a1f600aa34b97a77e47640bc-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:21,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=2, pid=33, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=794028a2b8fc0a516890763306629d4c, target=asf911.gq1.ygridcore.net,59970,1529623390066; rit=OPENING, location=asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,646 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(976): writing seq id for e04a2e293b4c1bbd21e87d569c11154e 2018-06-21 23:23:21,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=2, pid=25, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=8e2537011449fca6295a357cbde76e6c, target=asf911.gq1.ygridcore.net,59970,1529623390066; rit=OPENING, location=asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,646 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(923): replaying wal for 7480a111a1f600aa34b97a77e47640bc 2018-06-21 23:23:21,648 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(976): writing seq id for 7006d69958189f4e3b610c07f795d5d5 2018-06-21 23:23:21,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=2, pid=20, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=28b5f974b61112896f4c4f1127dc2b99, target=asf911.gq1.ygridcore.net,59970,1529623390066; rit=OPENING, location=asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,650 DEBUG [PostOpenDeployTasks:794028a2b8fc0a516890763306629d4c] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,zzz,1529623399386.794028a2b8fc0a516890763306629d4c. 2018-06-21 23:23:21,653 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] handler.OpenRegionHandler(128): Opened multi_test_table,zzz,1529623399386.794028a2b8fc0a516890763306629d4c. on asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,653 DEBUG [PostOpenDeployTasks:8e2537011449fca6295a357cbde76e6c] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c. 2018-06-21 23:23:21,653 DEBUG [PostOpenDeployTasks:28b5f974b61112896f4c4f1127dc2b99] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,mmm,1529623399386.28b5f974b61112896f4c4f1127dc2b99. 2018-06-21 23:23:21,654 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] handler.OpenRegionHandler(128): Opened multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c. on asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,654 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7094): Opening region: {ENCODED => 0dfb3ebb8c759d632ec793ac0387ec95, NAME => 'multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95.', STARTKEY => 'vvv', ENDKEY => 'www'} 2018-06-21 23:23:21,655 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] handler.OpenRegionHandler(128): Opened multi_test_table,mmm,1529623399386.28b5f974b61112896f4c4f1127dc2b99. on asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,656 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7094): Opening region: {ENCODED => 573e96916daa12dfc083e39a253dc3a2, NAME => 'multi_test_table,jjj,1529623399386.573e96916daa12dfc083e39a253dc3a2.', STARTKEY => 'jjj', ENDKEY => 'kkk'} 2018-06-21 23:23:21,656 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/7480a111a1f600aa34b97a77e47640bc 2018-06-21 23:23:21,656 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(931): stopping wal replay for 7480a111a1f600aa34b97a77e47640bc 2018-06-21 23:23:21,656 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:21,656 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(943): Cleaning up temporary data for 7480a111a1f600aa34b97a77e47640bc 2018-06-21 23:23:21,656 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(827): Instantiated multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:21,656 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table 573e96916daa12dfc083e39a253dc3a2 2018-06-21 23:23:21,657 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7134): checking encryption for 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:21,657 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(827): Instantiated multi_test_table,jjj,1529623399386.573e96916daa12dfc083e39a253dc3a2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:21,657 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7139): checking classloading for 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:21,657 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7134): checking encryption for 573e96916daa12dfc083e39a253dc3a2 2018-06-21 23:23:21,657 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7139): checking classloading for 573e96916daa12dfc083e39a253dc3a2 2018-06-21 23:23:21,658 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(954): Cleaning up detritus for 7480a111a1f600aa34b97a77e47640bc 2018-06-21 23:23:21,660 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/e04a2e293b4c1bbd21e87d569c11154e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-06-21 23:23:21,660 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/7006d69958189f4e3b610c07f795d5d5/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-06-21 23:23:21,661 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(976): writing seq id for 7480a111a1f600aa34b97a77e47640bc 2018-06-21 23:23:21,660 INFO [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(980): Opened e04a2e293b4c1bbd21e87d569c11154e; next sequenceid=2 2018-06-21 23:23:21,661 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(987): Running coprocessor post-open hooks for e04a2e293b4c1bbd21e87d569c11154e 2018-06-21 23:23:21,661 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(980): Opened 7006d69958189f4e3b610c07f795d5d5; next sequenceid=2 2018-06-21 23:23:21,661 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(987): Running coprocessor post-open hooks for 7006d69958189f4e3b610c07f795d5d5 2018-06-21 23:23:21,663 INFO [PostOpenDeployTasks:7006d69958189f4e3b610c07f795d5d5] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,yyy,1529623399386.7006d69958189f4e3b610c07f795d5d5. 2018-06-21 23:23:21,663 INFO [PostOpenDeployTasks:e04a2e293b4c1bbd21e87d569c11154e] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,bbb,1529623399386.e04a2e293b4c1bbd21e87d569c11154e. 2018-06-21 23:23:21,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=2, pid=32, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=7006d69958189f4e3b610c07f795d5d5, target=asf911.gq1.ygridcore.net,58235,1529623390382; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:21,666 DEBUG [StoreOpener-0dfb3ebb8c759d632ec793ac0387ec95-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/family 2018-06-21 23:23:21,667 DEBUG [StoreOpener-0dfb3ebb8c759d632ec793ac0387ec95-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/family 2018-06-21 23:23:21,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=2, pid=9, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=e04a2e293b4c1bbd21e87d569c11154e, target=asf911.gq1.ygridcore.net,58235,1529623390382; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:21,669 DEBUG [PostOpenDeployTasks:7006d69958189f4e3b610c07f795d5d5] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,yyy,1529623399386.7006d69958189f4e3b610c07f795d5d5. 2018-06-21 23:23:21,669 INFO [StoreOpener-0dfb3ebb8c759d632ec793ac0387ec95-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:21,670 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] handler.OpenRegionHandler(128): Opened multi_test_table,yyy,1529623399386.7006d69958189f4e3b610c07f795d5d5. on asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:21,670 DEBUG [PostOpenDeployTasks:e04a2e293b4c1bbd21e87d569c11154e] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,bbb,1529623399386.e04a2e293b4c1bbd21e87d569c11154e. 2018-06-21 23:23:21,671 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7094): Opening region: {ENCODED => cb0df54f72fae3cd39a03514dbbaa19c, NAME => 'multi_test_table,,1529623399386.cb0df54f72fae3cd39a03514dbbaa19c.', STARTKEY => '', ENDKEY => 'bbb'} 2018-06-21 23:23:21,671 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table cb0df54f72fae3cd39a03514dbbaa19c 2018-06-21 23:23:21,672 DEBUG [StoreOpener-573e96916daa12dfc083e39a253dc3a2-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/573e96916daa12dfc083e39a253dc3a2/family 2018-06-21 23:23:21,671 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] handler.OpenRegionHandler(128): Opened multi_test_table,bbb,1529623399386.e04a2e293b4c1bbd21e87d569c11154e. on asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:21,671 INFO [StoreOpener-0dfb3ebb8c759d632ec793ac0387ec95-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:21,672 DEBUG [StoreOpener-573e96916daa12dfc083e39a253dc3a2-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/573e96916daa12dfc083e39a253dc3a2/family 2018-06-21 23:23:21,672 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(827): Instantiated multi_test_table,,1529623399386.cb0df54f72fae3cd39a03514dbbaa19c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:21,672 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7134): checking encryption for cb0df54f72fae3cd39a03514dbbaa19c 2018-06-21 23:23:21,673 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7139): checking classloading for cb0df54f72fae3cd39a03514dbbaa19c 2018-06-21 23:23:21,673 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/7480a111a1f600aa34b97a77e47640bc/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-06-21 23:23:21,673 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(980): Opened 7480a111a1f600aa34b97a77e47640bc; next sequenceid=2 2018-06-21 23:23:21,673 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(987): Running coprocessor post-open hooks for 7480a111a1f600aa34b97a77e47640bc 2018-06-21 23:23:21,673 INFO [StoreOpener-0dfb3ebb8c759d632ec793ac0387ec95-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:21,673 INFO [StoreOpener-573e96916daa12dfc083e39a253dc3a2-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:21,675 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(923): replaying wal for 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:21,676 INFO [StoreOpener-573e96916daa12dfc083e39a253dc3a2-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:21,678 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:21,678 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(931): stopping wal replay for 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:21,678 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(943): Cleaning up temporary data for 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:21,680 INFO [StoreOpener-573e96916daa12dfc083e39a253dc3a2-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:21,685 INFO [PostOpenDeployTasks:7480a111a1f600aa34b97a77e47640bc] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,qqq,1529623399386.7480a111a1f600aa34b97a77e47640bc. 2018-06-21 23:23:21,685 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(923): replaying wal for 573e96916daa12dfc083e39a253dc3a2 2018-06-21 23:23:21,688 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(954): Cleaning up detritus for 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:21,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=2, pid=24, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=7480a111a1f600aa34b97a77e47640bc, target=asf911.gq1.ygridcore.net,58235,1529623390382; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:21,690 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/573e96916daa12dfc083e39a253dc3a2 2018-06-21 23:23:21,690 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(931): stopping wal replay for 573e96916daa12dfc083e39a253dc3a2 2018-06-21 23:23:21,690 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(943): Cleaning up temporary data for 573e96916daa12dfc083e39a253dc3a2 2018-06-21 23:23:21,691 DEBUG [PostOpenDeployTasks:7480a111a1f600aa34b97a77e47640bc] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,qqq,1529623399386.7480a111a1f600aa34b97a77e47640bc. 2018-06-21 23:23:21,692 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] handler.OpenRegionHandler(128): Opened multi_test_table,qqq,1529623399386.7480a111a1f600aa34b97a77e47640bc. on asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:21,692 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(954): Cleaning up detritus for 573e96916daa12dfc083e39a253dc3a2 2018-06-21 23:23:21,693 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(976): writing seq id for 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:21,694 DEBUG [StoreOpener-cb0df54f72fae3cd39a03514dbbaa19c-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/cb0df54f72fae3cd39a03514dbbaa19c/family 2018-06-21 23:23:21,694 DEBUG [StoreOpener-cb0df54f72fae3cd39a03514dbbaa19c-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/cb0df54f72fae3cd39a03514dbbaa19c/family 2018-06-21 23:23:21,695 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(976): writing seq id for 573e96916daa12dfc083e39a253dc3a2 2018-06-21 23:23:21,695 INFO [StoreOpener-cb0df54f72fae3cd39a03514dbbaa19c-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:21,697 DEBUG [PEWorker-13] assignment.RegionTransitionProcedure(354): Finishing pid=24, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=7480a111a1f600aa34b97a77e47640bc, target=asf911.gq1.ygridcore.net,58235,1529623390382; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:21,697 DEBUG [PEWorker-6] assignment.RegionTransitionProcedure(354): Finishing pid=9, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=e04a2e293b4c1bbd21e87d569c11154e, target=asf911.gq1.ygridcore.net,58235,1529623390382; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:21,697 DEBUG [PEWorker-11] assignment.RegionTransitionProcedure(354): Finishing pid=25, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=8e2537011449fca6295a357cbde76e6c, target=asf911.gq1.ygridcore.net,59970,1529623390066; rit=OPENING, location=asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,697 DEBUG [PEWorker-15] assignment.RegionTransitionProcedure(354): Finishing pid=33, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=794028a2b8fc0a516890763306629d4c, target=asf911.gq1.ygridcore.net,59970,1529623390066; rit=OPENING, location=asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,699 INFO [PEWorker-2] assignment.RegionStateStore(198): pid=15 updating hbase:meta row=91ab16f39c84b448bdc2e5a1e1fc801b, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:21,699 INFO [PEWorker-8] assignment.RegionStateStore(198): pid=23 updating hbase:meta row=36d0a953419e9f81636aebf73e7ed5df, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:21,699 INFO [PEWorker-1] assignment.RegionStateStore(198): pid=26 updating hbase:meta row=d4ac3b44d02d68507d6317e201672011, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:21,697 DEBUG [PEWorker-10] assignment.RegionTransitionProcedure(354): Finishing pid=32, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=7006d69958189f4e3b610c07f795d5d5, target=asf911.gq1.ygridcore.net,58235,1529623390382; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:21,697 DEBUG [PEWorker-12] assignment.RegionTransitionProcedure(354): Finishing pid=20, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=28b5f974b61112896f4c4f1127dc2b99, target=asf911.gq1.ygridcore.net,59970,1529623390066; rit=OPENING, location=asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,700 INFO [PEWorker-6] assignment.RegionStateStore(198): pid=9 updating hbase:meta row=e04a2e293b4c1bbd21e87d569c11154e, regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:21,699 INFO [PEWorker-15] assignment.RegionStateStore(198): pid=33 updating hbase:meta row=794028a2b8fc0a516890763306629d4c, regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,699 INFO [PEWorker-13] assignment.RegionStateStore(198): pid=24 updating hbase:meta row=7480a111a1f600aa34b97a77e47640bc, regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:21,699 INFO [StoreOpener-cb0df54f72fae3cd39a03514dbbaa19c-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:21,699 INFO [PEWorker-11] assignment.RegionStateStore(198): pid=25 updating hbase:meta row=8e2537011449fca6295a357cbde76e6c, regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,699 INFO [PEWorker-3] assignment.RegionStateStore(198): pid=27 updating hbase:meta row=dff6ab063ea78552b5e3c3ae7760ca5a, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:21,699 INFO [PEWorker-5] assignment.RegionStateStore(198): pid=12 updating hbase:meta row=c793c2c3948aad84865d30ebf71a529e, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:21,699 INFO [PEWorker-4] assignment.RegionStateStore(198): pid=19 updating hbase:meta row=77885b472341ce945c76aa06a4627146, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:21,699 INFO [PEWorker-16] assignment.RegionStateStore(198): pid=10 updating hbase:meta row=124da0ae361b264a91ea989bcd12b459, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:21,699 INFO [PEWorker-7] assignment.RegionStateStore(198): pid=11 updating hbase:meta row=e66f718b26d1c4420497ccfb42f08321, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:21,702 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/573e96916daa12dfc083e39a253dc3a2/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-06-21 23:23:21,703 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(980): Opened 573e96916daa12dfc083e39a253dc3a2; next sequenceid=2 2018-06-21 23:23:21,703 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(987): Running coprocessor post-open hooks for 573e96916daa12dfc083e39a253dc3a2 2018-06-21 23:23:21,700 INFO [PEWorker-12] assignment.RegionStateStore(198): pid=20 updating hbase:meta row=28b5f974b61112896f4c4f1127dc2b99, regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,700 INFO [PEWorker-10] assignment.RegionStateStore(198): pid=32 updating hbase:meta row=7006d69958189f4e3b610c07f795d5d5, regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:21,703 INFO [StoreOpener-cb0df54f72fae3cd39a03514dbbaa19c-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:21,703 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-06-21 23:23:21,705 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(923): replaying wal for cb0df54f72fae3cd39a03514dbbaa19c 2018-06-21 23:23:21,706 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(119): Fallback to compat rpc execution for serverName=asf911.gq1.ygridcore.net,55146,1529623388534 version=0 2018-06-21 23:23:21,706 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(980): Opened 0dfb3ebb8c759d632ec793ac0387ec95; next sequenceid=2 2018-06-21 23:23:21,707 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(987): Running coprocessor post-open hooks for 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:21,708 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/cb0df54f72fae3cd39a03514dbbaa19c 2018-06-21 23:23:21,708 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(931): stopping wal replay for cb0df54f72fae3cd39a03514dbbaa19c 2018-06-21 23:23:21,708 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(943): Cleaning up temporary data for cb0df54f72fae3cd39a03514dbbaa19c 2018-06-21 23:23:21,707 INFO [PostOpenDeployTasks:573e96916daa12dfc083e39a253dc3a2] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,jjj,1529623399386.573e96916daa12dfc083e39a253dc3a2. 2018-06-21 23:23:21,708 INFO [PEWorker-2] assignment.RegionTransitionProcedure(241): Dispatch pid=15, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=91ab16f39c84b448bdc2e5a1e1fc801b, target=asf911.gq1.ygridcore.net,40767,1529623390148; rit=OPENING, location=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:21,709 INFO [PEWorker-8] assignment.RegionTransitionProcedure(241): Dispatch pid=23, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=36d0a953419e9f81636aebf73e7ed5df, target=asf911.gq1.ygridcore.net,40767,1529623390148; rit=OPENING, location=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:21,711 INFO [PostOpenDeployTasks:0dfb3ebb8c759d632ec793ac0387ec95] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95. 2018-06-21 23:23:21,712 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(954): Cleaning up detritus for cb0df54f72fae3cd39a03514dbbaa19c 2018-06-21 23:23:21,712 INFO [PEWorker-1] assignment.RegionTransitionProcedure(241): Dispatch pid=26, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=d4ac3b44d02d68507d6317e201672011, target=asf911.gq1.ygridcore.net,40767,1529623390148; rit=OPENING, location=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:21,714 DEBUG [RSProcedureDispatcher-pool3-t7] master.ServerManager(746): New admin connection to asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:21,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=2, pid=17, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=573e96916daa12dfc083e39a253dc3a2, target=asf911.gq1.ygridcore.net,59970,1529623390066; rit=OPENING, location=asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,712 DEBUG [RSProcedureDispatcher-pool3-t6] master.ServerManager(746): New admin connection to asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:21,716 INFO [RSProcedureDispatcher-pool3-t7] regionserver.RSRpcServices(1980): Open multi_test_table,xxx,1529623399386.77a32d153e5d8148ca9d599c6a3cb695. 2018-06-21 23:23:21,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=2, pid=29, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=0dfb3ebb8c759d632ec793ac0387ec95, target=asf911.gq1.ygridcore.net,59970,1529623390066; rit=OPENING, location=asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,716 DEBUG [PostOpenDeployTasks:573e96916daa12dfc083e39a253dc3a2] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,jjj,1529623399386.573e96916daa12dfc083e39a253dc3a2. 2018-06-21 23:23:21,718 DEBUG [PostOpenDeployTasks:0dfb3ebb8c759d632ec793ac0387ec95] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95. 2018-06-21 23:23:21,719 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] handler.OpenRegionHandler(128): Opened multi_test_table,jjj,1529623399386.573e96916daa12dfc083e39a253dc3a2. on asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,720 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] handler.OpenRegionHandler(128): Opened multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95. on asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,720 INFO [RSProcedureDispatcher-pool3-t7] regionserver.RSRpcServices(1980): Open multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173. 2018-06-21 23:23:21,721 DEBUG [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegion(7094): Opening region: {ENCODED => 77a32d153e5d8148ca9d599c6a3cb695, NAME => 'multi_test_table,xxx,1529623399386.77a32d153e5d8148ca9d599c6a3cb695.', STARTKEY => 'xxx', ENDKEY => 'yyy'} 2018-06-21 23:23:21,722 DEBUG [RS_OPEN_REGION-master/asf911:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table 77a32d153e5d8148ca9d599c6a3cb695 2018-06-21 23:23:21,723 DEBUG [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegion(827): Instantiated multi_test_table,xxx,1529623399386.77a32d153e5d8148ca9d599c6a3cb695.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:21,723 DEBUG [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegion(7134): checking encryption for 77a32d153e5d8148ca9d599c6a3cb695 2018-06-21 23:23:21,723 INFO [RSProcedureDispatcher-pool3-t7] regionserver.RSRpcServices(1980): Open multi_test_table,nnn,1529623399386.b1a8d69ffdd43e80f30a919effdc7c4e. 2018-06-21 23:23:21,725 INFO [RS-EventLoopGroup-6-2] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-06-21 23:23:21,725 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(976): writing seq id for cb0df54f72fae3cd39a03514dbbaa19c 2018-06-21 23:23:21,725 INFO [RS-EventLoopGroup-6-2] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 56446 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:21,725 DEBUG [RS_OPEN_REGION-master/asf911:0-1] regionserver.HRegion(7094): Opening region: {ENCODED => fee67ae5d4f2b2be35fa516877a8d173, NAME => 'multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173.', STARTKEY => 'kkk', ENDKEY => 'lll'} 2018-06-21 23:23:21,723 DEBUG [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegion(7139): checking classloading for 77a32d153e5d8148ca9d599c6a3cb695 2018-06-21 23:23:21,727 DEBUG [RS_OPEN_REGION-master/asf911:0-2] regionserver.HRegion(7094): Opening region: {ENCODED => b1a8d69ffdd43e80f30a919effdc7c4e, NAME => 'multi_test_table,nnn,1529623399386.b1a8d69ffdd43e80f30a919effdc7c4e.', STARTKEY => 'nnn', ENDKEY => 'ooo'} 2018-06-21 23:23:21,725 INFO [RSProcedureDispatcher-pool3-t7] regionserver.RSRpcServices(1980): Open multi_test_table,iii,1529623399386.04d71925444c7c823938ba3d9ad0504d. 2018-06-21 23:23:21,728 DEBUG [RS_OPEN_REGION-master/asf911:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:21,728 DEBUG [RS_OPEN_REGION-master/asf911:0-1] regionserver.HRegion(827): Instantiated multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:21,728 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=56613] regionserver.RSRpcServices(1980): Open multi_test_table,ooo,1529623399386.1449fc3d526b1c6ab2fd158edec23277. 2018-06-21 23:23:21,728 DEBUG [RS_OPEN_REGION-master/asf911:0-1] regionserver.HRegion(7134): checking encryption for fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:21,728 DEBUG [RS_OPEN_REGION-master/asf911:0-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table b1a8d69ffdd43e80f30a919effdc7c4e 2018-06-21 23:23:21,730 DEBUG [RS_OPEN_REGION-master/asf911:0-2] regionserver.HRegion(827): Instantiated multi_test_table,nnn,1529623399386.b1a8d69ffdd43e80f30a919effdc7c4e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:21,731 DEBUG [RS_OPEN_REGION-master/asf911:0-2] regionserver.HRegion(7134): checking encryption for b1a8d69ffdd43e80f30a919effdc7c4e 2018-06-21 23:23:21,731 DEBUG [RS_OPEN_REGION-master/asf911:0-2] regionserver.HRegion(7139): checking classloading for b1a8d69ffdd43e80f30a919effdc7c4e 2018-06-21 23:23:21,730 DEBUG [RS_OPEN_REGION-master/asf911:0-1] regionserver.HRegion(7139): checking classloading for fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:21,732 INFO [PEWorker-3] assignment.RegionTransitionProcedure(241): Dispatch pid=27, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=dff6ab063ea78552b5e3c3ae7760ca5a, target=asf911.gq1.ygridcore.net,52507,1529623390228; rit=OPENING, location=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:21,732 INFO [PEWorker-4] assignment.RegionTransitionProcedure(241): Dispatch pid=19, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=77885b472341ce945c76aa06a4627146, target=asf911.gq1.ygridcore.net,52507,1529623390228; rit=OPENING, location=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:21,732 INFO [PEWorker-16] assignment.RegionTransitionProcedure(241): Dispatch pid=10, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=124da0ae361b264a91ea989bcd12b459, target=asf911.gq1.ygridcore.net,52507,1529623390228; rit=OPENING, location=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:21,735 INFO [PEWorker-5] assignment.RegionTransitionProcedure(241): Dispatch pid=12, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=c793c2c3948aad84865d30ebf71a529e, target=asf911.gq1.ygridcore.net,40767,1529623390148; rit=OPENING, location=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:21,736 INFO [PEWorker-7] assignment.RegionTransitionProcedure(241): Dispatch pid=11, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=e66f718b26d1c4420497ccfb42f08321, target=asf911.gq1.ygridcore.net,52507,1529623390228; rit=OPENING, location=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:21,739 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7094): Opening region: {ENCODED => 1449fc3d526b1c6ab2fd158edec23277, NAME => 'multi_test_table,ooo,1529623399386.1449fc3d526b1c6ab2fd158edec23277.', STARTKEY => 'ooo', ENDKEY => 'ppp'} 2018-06-21 23:23:21,739 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=56613] regionserver.RSRpcServices(1980): Open multi_test_table,ggg,1529623399386.10cdebbb720d98123807841a0febf405. 2018-06-21 23:23:21,740 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table 1449fc3d526b1c6ab2fd158edec23277 2018-06-21 23:23:21,740 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(827): Instantiated multi_test_table,ooo,1529623399386.1449fc3d526b1c6ab2fd158edec23277.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:21,740 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=56613] regionserver.RSRpcServices(1980): Open multi_test_table,www,1529623399386.a657e5e90b9697e4cf81bc4bbde44800. 2018-06-21 23:23:21,745 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7094): Opening region: {ENCODED => 10cdebbb720d98123807841a0febf405, NAME => 'multi_test_table,ggg,1529623399386.10cdebbb720d98123807841a0febf405.', STARTKEY => 'ggg', ENDKEY => 'hhh'} 2018-06-21 23:23:21,745 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7134): checking encryption for 1449fc3d526b1c6ab2fd158edec23277 2018-06-21 23:23:21,746 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=56613] regionserver.RSRpcServices(1980): Open multi_test_table,fff,1529623399386.da9d0eefced1af8dfad2324a5d9e1bf7. 2018-06-21 23:23:21,746 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7139): checking classloading for 1449fc3d526b1c6ab2fd158edec23277 2018-06-21 23:23:21,746 DEBUG [StoreOpener-b1a8d69ffdd43e80f30a919effdc7c4e-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/b1a8d69ffdd43e80f30a919effdc7c4e/family 2018-06-21 23:23:21,747 DEBUG [StoreOpener-b1a8d69ffdd43e80f30a919effdc7c4e-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/b1a8d69ffdd43e80f30a919effdc7c4e/family 2018-06-21 23:23:21,747 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7094): Opening region: {ENCODED => a657e5e90b9697e4cf81bc4bbde44800, NAME => 'multi_test_table,www,1529623399386.a657e5e90b9697e4cf81bc4bbde44800.', STARTKEY => 'www', ENDKEY => 'xxx'} 2018-06-21 23:23:21,749 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table 10cdebbb720d98123807841a0febf405 2018-06-21 23:23:21,749 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/cb0df54f72fae3cd39a03514dbbaa19c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-06-21 23:23:21,749 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table a657e5e90b9697e4cf81bc4bbde44800 2018-06-21 23:23:21,749 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(827): Instantiated multi_test_table,ggg,1529623399386.10cdebbb720d98123807841a0febf405.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:21,749 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(980): Opened cb0df54f72fae3cd39a03514dbbaa19c; next sequenceid=2 2018-06-21 23:23:21,750 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(987): Running coprocessor post-open hooks for cb0df54f72fae3cd39a03514dbbaa19c 2018-06-21 23:23:21,749 DEBUG [StoreOpener-77a32d153e5d8148ca9d599c6a3cb695-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/77a32d153e5d8148ca9d599c6a3cb695/family 2018-06-21 23:23:21,750 DEBUG [StoreOpener-77a32d153e5d8148ca9d599c6a3cb695-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/77a32d153e5d8148ca9d599c6a3cb695/family 2018-06-21 23:23:21,750 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7134): checking encryption for 10cdebbb720d98123807841a0febf405 2018-06-21 23:23:21,749 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(827): Instantiated multi_test_table,www,1529623399386.a657e5e90b9697e4cf81bc4bbde44800.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:21,749 INFO [StoreOpener-b1a8d69ffdd43e80f30a919effdc7c4e-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:21,751 DEBUG [StoreOpener-fee67ae5d4f2b2be35fa516877a8d173-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/family 2018-06-21 23:23:21,751 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7134): checking encryption for a657e5e90b9697e4cf81bc4bbde44800 2018-06-21 23:23:21,750 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7139): checking classloading for 10cdebbb720d98123807841a0febf405 2018-06-21 23:23:21,751 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7139): checking classloading for a657e5e90b9697e4cf81bc4bbde44800 2018-06-21 23:23:21,751 DEBUG [StoreOpener-fee67ae5d4f2b2be35fa516877a8d173-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/family 2018-06-21 23:23:21,751 INFO [StoreOpener-b1a8d69ffdd43e80f30a919effdc7c4e-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:21,751 INFO [StoreOpener-77a32d153e5d8148ca9d599c6a3cb695-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:21,752 INFO [StoreOpener-fee67ae5d4f2b2be35fa516877a8d173-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:21,752 INFO [PostOpenDeployTasks:cb0df54f72fae3cd39a03514dbbaa19c] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,,1529623399386.cb0df54f72fae3cd39a03514dbbaa19c. 2018-06-21 23:23:21,752 INFO [StoreOpener-77a32d153e5d8148ca9d599c6a3cb695-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:21,753 INFO [StoreOpener-fee67ae5d4f2b2be35fa516877a8d173-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:21,753 INFO [StoreOpener-b1a8d69ffdd43e80f30a919effdc7c4e-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:21,753 INFO [StoreOpener-77a32d153e5d8148ca9d599c6a3cb695-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:21,754 INFO [StoreOpener-fee67ae5d4f2b2be35fa516877a8d173-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:21,754 DEBUG [RS_OPEN_REGION-master/asf911:0-2] regionserver.HRegion(923): replaying wal for b1a8d69ffdd43e80f30a919effdc7c4e 2018-06-21 23:23:21,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=2, pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=cb0df54f72fae3cd39a03514dbbaa19c, target=asf911.gq1.ygridcore.net,58235,1529623390382; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:21,757 DEBUG [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegion(923): replaying wal for 77a32d153e5d8148ca9d599c6a3cb695 2018-06-21 23:23:21,757 DEBUG [RS_OPEN_REGION-master/asf911:0-2] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/b1a8d69ffdd43e80f30a919effdc7c4e 2018-06-21 23:23:21,757 DEBUG [RS_OPEN_REGION-master/asf911:0-2] regionserver.HRegion(931): stopping wal replay for b1a8d69ffdd43e80f30a919effdc7c4e 2018-06-21 23:23:21,757 DEBUG [RS_OPEN_REGION-master/asf911:0-2] regionserver.HRegion(943): Cleaning up temporary data for b1a8d69ffdd43e80f30a919effdc7c4e 2018-06-21 23:23:21,758 DEBUG [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/77a32d153e5d8148ca9d599c6a3cb695 2018-06-21 23:23:21,758 DEBUG [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegion(931): stopping wal replay for 77a32d153e5d8148ca9d599c6a3cb695 2018-06-21 23:23:21,758 DEBUG [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegion(943): Cleaning up temporary data for 77a32d153e5d8148ca9d599c6a3cb695 2018-06-21 23:23:21,759 DEBUG [RS_OPEN_REGION-master/asf911:0-2] regionserver.HRegion(954): Cleaning up detritus for b1a8d69ffdd43e80f30a919effdc7c4e 2018-06-21 23:23:21,759 DEBUG [StoreOpener-10cdebbb720d98123807841a0febf405-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/10cdebbb720d98123807841a0febf405/family 2018-06-21 23:23:21,759 DEBUG [PostOpenDeployTasks:cb0df54f72fae3cd39a03514dbbaa19c] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,,1529623399386.cb0df54f72fae3cd39a03514dbbaa19c. 2018-06-21 23:23:21,759 DEBUG [RS_OPEN_REGION-master/asf911:0-1] regionserver.HRegion(923): replaying wal for fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:21,759 DEBUG [StoreOpener-10cdebbb720d98123807841a0febf405-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/10cdebbb720d98123807841a0febf405/family 2018-06-21 23:23:21,761 DEBUG [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegion(954): Cleaning up detritus for 77a32d153e5d8148ca9d599c6a3cb695 2018-06-21 23:23:21,761 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] handler.OpenRegionHandler(128): Opened multi_test_table,,1529623399386.cb0df54f72fae3cd39a03514dbbaa19c. on asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:21,761 INFO [StoreOpener-10cdebbb720d98123807841a0febf405-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:21,762 DEBUG [RS_OPEN_REGION-master/asf911:0-1] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:21,762 DEBUG [RS_OPEN_REGION-master/asf911:0-1] regionserver.HRegion(931): stopping wal replay for fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:21,762 DEBUG [RS_OPEN_REGION-master/asf911:0-1] regionserver.HRegion(943): Cleaning up temporary data for fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:21,762 INFO [StoreOpener-10cdebbb720d98123807841a0febf405-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:21,763 DEBUG [RS_OPEN_REGION-master/asf911:0-1] regionserver.HRegion(954): Cleaning up detritus for fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:21,763 INFO [StoreOpener-10cdebbb720d98123807841a0febf405-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:21,763 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(923): replaying wal for 10cdebbb720d98123807841a0febf405 2018-06-21 23:23:21,764 DEBUG [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegion(976): writing seq id for 77a32d153e5d8148ca9d599c6a3cb695 2018-06-21 23:23:21,764 DEBUG [RS_OPEN_REGION-master/asf911:0-2] regionserver.HRegion(976): writing seq id for b1a8d69ffdd43e80f30a919effdc7c4e 2018-06-21 23:23:21,764 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/10cdebbb720d98123807841a0febf405 2018-06-21 23:23:21,770 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(931): stopping wal replay for 10cdebbb720d98123807841a0febf405 2018-06-21 23:23:21,770 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(943): Cleaning up temporary data for 10cdebbb720d98123807841a0febf405 2018-06-21 23:23:21,771 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(954): Cleaning up detritus for 10cdebbb720d98123807841a0febf405 2018-06-21 23:23:21,771 DEBUG [RS_OPEN_REGION-master/asf911:0-1] regionserver.HRegion(976): writing seq id for fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:21,776 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(976): writing seq id for 10cdebbb720d98123807841a0febf405 2018-06-21 23:23:21,779 DEBUG [StoreOpener-a657e5e90b9697e4cf81bc4bbde44800-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/a657e5e90b9697e4cf81bc4bbde44800/family 2018-06-21 23:23:21,779 DEBUG [StoreOpener-a657e5e90b9697e4cf81bc4bbde44800-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/a657e5e90b9697e4cf81bc4bbde44800/family 2018-06-21 23:23:21,779 DEBUG [StoreOpener-1449fc3d526b1c6ab2fd158edec23277-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/1449fc3d526b1c6ab2fd158edec23277/family 2018-06-21 23:23:21,779 DEBUG [StoreOpener-1449fc3d526b1c6ab2fd158edec23277-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/1449fc3d526b1c6ab2fd158edec23277/family 2018-06-21 23:23:21,780 INFO [StoreOpener-a657e5e90b9697e4cf81bc4bbde44800-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:21,782 INFO [StoreOpener-a657e5e90b9697e4cf81bc4bbde44800-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:21,782 INFO [StoreOpener-1449fc3d526b1c6ab2fd158edec23277-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:21,783 DEBUG [RS_OPEN_REGION-master/asf911:0-1] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-06-21 23:23:21,783 INFO [RS_OPEN_REGION-master/asf911:0-1] regionserver.HRegion(980): Opened fee67ae5d4f2b2be35fa516877a8d173; next sequenceid=2 2018-06-21 23:23:21,783 DEBUG [RS_OPEN_REGION-master/asf911:0-1] regionserver.HRegion(987): Running coprocessor post-open hooks for fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:21,783 INFO [StoreOpener-1449fc3d526b1c6ab2fd158edec23277-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:21,783 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/10cdebbb720d98123807841a0febf405/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-06-21 23:23:21,783 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(980): Opened 10cdebbb720d98123807841a0febf405; next sequenceid=2 2018-06-21 23:23:21,784 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(987): Running coprocessor post-open hooks for 10cdebbb720d98123807841a0febf405 2018-06-21 23:23:21,784 DEBUG [RS_OPEN_REGION-master/asf911:0-0] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/77a32d153e5d8148ca9d599c6a3cb695/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-06-21 23:23:21,784 INFO [StoreOpener-a657e5e90b9697e4cf81bc4bbde44800-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:21,784 DEBUG [RS_OPEN_REGION-master/asf911:0-2] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/b1a8d69ffdd43e80f30a919effdc7c4e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-06-21 23:23:21,784 INFO [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegion(980): Opened 77a32d153e5d8148ca9d599c6a3cb695; next sequenceid=2 2018-06-21 23:23:21,784 DEBUG [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegion(987): Running coprocessor post-open hooks for 77a32d153e5d8148ca9d599c6a3cb695 2018-06-21 23:23:21,784 INFO [RS_OPEN_REGION-master/asf911:0-2] regionserver.HRegion(980): Opened b1a8d69ffdd43e80f30a919effdc7c4e; next sequenceid=2 2018-06-21 23:23:21,784 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(923): replaying wal for a657e5e90b9697e4cf81bc4bbde44800 2018-06-21 23:23:21,784 INFO [StoreOpener-1449fc3d526b1c6ab2fd158edec23277-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:21,792 DEBUG [RS_OPEN_REGION-master/asf911:0-2] regionserver.HRegion(987): Running coprocessor post-open hooks for b1a8d69ffdd43e80f30a919effdc7c4e 2018-06-21 23:23:21,792 INFO [PostOpenDeployTasks:fee67ae5d4f2b2be35fa516877a8d173] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173. 2018-06-21 23:23:21,792 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(923): replaying wal for 1449fc3d526b1c6ab2fd158edec23277 2018-06-21 23:23:21,796 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/a657e5e90b9697e4cf81bc4bbde44800 2018-06-21 23:23:21,796 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(931): stopping wal replay for a657e5e90b9697e4cf81bc4bbde44800 2018-06-21 23:23:21,796 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(943): Cleaning up temporary data for a657e5e90b9697e4cf81bc4bbde44800 2018-06-21 23:23:21,799 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/1449fc3d526b1c6ab2fd158edec23277 2018-06-21 23:23:21,799 DEBUG [PostOpenDeployTasks:fee67ae5d4f2b2be35fa516877a8d173] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=2, pid=18, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=fee67ae5d4f2b2be35fa516877a8d173, target=asf911.gq1.ygridcore.net,55146,1529623388534; rit=OPENING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:21,799 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(931): stopping wal replay for 1449fc3d526b1c6ab2fd158edec23277 2018-06-21 23:23:21,799 INFO [PostOpenDeployTasks:77a32d153e5d8148ca9d599c6a3cb695] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,xxx,1529623399386.77a32d153e5d8148ca9d599c6a3cb695. 2018-06-21 23:23:21,802 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(954): Cleaning up detritus for a657e5e90b9697e4cf81bc4bbde44800 2018-06-21 23:23:21,801 INFO [PostOpenDeployTasks:10cdebbb720d98123807841a0febf405] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,ggg,1529623399386.10cdebbb720d98123807841a0febf405. 2018-06-21 23:23:21,799 DEBUG [PostOpenDeployTasks:fee67ae5d4f2b2be35fa516877a8d173] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173. 2018-06-21 23:23:21,802 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(943): Cleaning up temporary data for 1449fc3d526b1c6ab2fd158edec23277 2018-06-21 23:23:21,809 DEBUG [RS_OPEN_REGION-master/asf911:0-1] handler.OpenRegionHandler(128): Opened multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173. on asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:21,809 DEBUG [RS_OPEN_REGION-master/asf911:0-1] regionserver.HRegion(7094): Opening region: {ENCODED => 04d71925444c7c823938ba3d9ad0504d, NAME => 'multi_test_table,iii,1529623399386.04d71925444c7c823938ba3d9ad0504d.', STARTKEY => 'iii', ENDKEY => 'jjj'} 2018-06-21 23:23:21,810 DEBUG [PostOpenDeployTasks:77a32d153e5d8148ca9d599c6a3cb695] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=2, pid=31, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=77a32d153e5d8148ca9d599c6a3cb695, target=asf911.gq1.ygridcore.net,55146,1529623388534; rit=OPENING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:21,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=2, pid=14, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=10cdebbb720d98123807841a0febf405, target=asf911.gq1.ygridcore.net,56613,1529623390297; rit=OPENING, location=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:21,814 DEBUG [PostOpenDeployTasks:77a32d153e5d8148ca9d599c6a3cb695] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,xxx,1529623399386.77a32d153e5d8148ca9d599c6a3cb695. 2018-06-21 23:23:21,814 DEBUG [RS_OPEN_REGION-master/asf911:0-0] handler.OpenRegionHandler(128): Opened multi_test_table,xxx,1529623399386.77a32d153e5d8148ca9d599c6a3cb695. on asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:21,814 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(954): Cleaning up detritus for 1449fc3d526b1c6ab2fd158edec23277 2018-06-21 23:23:21,814 DEBUG [RS_OPEN_REGION-master/asf911:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table 04d71925444c7c823938ba3d9ad0504d 2018-06-21 23:23:21,814 DEBUG [PostOpenDeployTasks:10cdebbb720d98123807841a0febf405] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,ggg,1529623399386.10cdebbb720d98123807841a0febf405. 2018-06-21 23:23:21,814 DEBUG [RS_OPEN_REGION-master/asf911:0-1] regionserver.HRegion(827): Instantiated multi_test_table,iii,1529623399386.04d71925444c7c823938ba3d9ad0504d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:21,815 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] handler.OpenRegionHandler(128): Opened multi_test_table,ggg,1529623399386.10cdebbb720d98123807841a0febf405. on asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:21,816 DEBUG [RS_OPEN_REGION-master/asf911:0-1] regionserver.HRegion(7134): checking encryption for 04d71925444c7c823938ba3d9ad0504d 2018-06-21 23:23:21,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=55146] master.MasterRpcServices(1142): Checking to see if procedure is done pid=7 2018-06-21 23:23:21,816 DEBUG [RS_OPEN_REGION-master/asf911:0-1] regionserver.HRegion(7139): checking classloading for 04d71925444c7c823938ba3d9ad0504d 2018-06-21 23:23:21,816 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7094): Opening region: {ENCODED => da9d0eefced1af8dfad2324a5d9e1bf7, NAME => 'multi_test_table,fff,1529623399386.da9d0eefced1af8dfad2324a5d9e1bf7.', STARTKEY => 'fff', ENDKEY => 'ggg'} 2018-06-21 23:23:21,816 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(976): writing seq id for a657e5e90b9697e4cf81bc4bbde44800 2018-06-21 23:23:21,817 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table da9d0eefced1af8dfad2324a5d9e1bf7 2018-06-21 23:23:21,817 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(827): Instantiated multi_test_table,fff,1529623399386.da9d0eefced1af8dfad2324a5d9e1bf7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:21,817 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7134): checking encryption for da9d0eefced1af8dfad2324a5d9e1bf7 2018-06-21 23:23:21,817 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(976): writing seq id for 1449fc3d526b1c6ab2fd158edec23277 2018-06-21 23:23:21,817 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7139): checking classloading for da9d0eefced1af8dfad2324a5d9e1bf7 2018-06-21 23:23:21,830 INFO [PostOpenDeployTasks:b1a8d69ffdd43e80f30a919effdc7c4e] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,nnn,1529623399386.b1a8d69ffdd43e80f30a919effdc7c4e. 2018-06-21 23:23:21,830 DEBUG [StoreOpener-da9d0eefced1af8dfad2324a5d9e1bf7-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/da9d0eefced1af8dfad2324a5d9e1bf7/family 2018-06-21 23:23:21,832 DEBUG [StoreOpener-da9d0eefced1af8dfad2324a5d9e1bf7-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/da9d0eefced1af8dfad2324a5d9e1bf7/family 2018-06-21 23:23:21,832 DEBUG [PostOpenDeployTasks:b1a8d69ffdd43e80f30a919effdc7c4e] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=2, pid=21, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=b1a8d69ffdd43e80f30a919effdc7c4e, target=asf911.gq1.ygridcore.net,55146,1529623388534; rit=OPENING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:21,832 DEBUG [StoreOpener-04d71925444c7c823938ba3d9ad0504d-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/04d71925444c7c823938ba3d9ad0504d/family 2018-06-21 23:23:21,833 DEBUG [StoreOpener-04d71925444c7c823938ba3d9ad0504d-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/04d71925444c7c823938ba3d9ad0504d/family 2018-06-21 23:23:21,832 DEBUG [PostOpenDeployTasks:b1a8d69ffdd43e80f30a919effdc7c4e] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,nnn,1529623399386.b1a8d69ffdd43e80f30a919effdc7c4e. 2018-06-21 23:23:21,833 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/a657e5e90b9697e4cf81bc4bbde44800/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-06-21 23:23:21,833 INFO [StoreOpener-da9d0eefced1af8dfad2324a5d9e1bf7-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:21,834 DEBUG [RS_OPEN_REGION-master/asf911:0-2] handler.OpenRegionHandler(128): Opened multi_test_table,nnn,1529623399386.b1a8d69ffdd43e80f30a919effdc7c4e. on asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:21,833 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/1449fc3d526b1c6ab2fd158edec23277/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-06-21 23:23:21,835 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(980): Opened 1449fc3d526b1c6ab2fd158edec23277; next sequenceid=2 2018-06-21 23:23:21,834 INFO [StoreOpener-04d71925444c7c823938ba3d9ad0504d-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:21,834 INFO [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(980): Opened a657e5e90b9697e4cf81bc4bbde44800; next sequenceid=2 2018-06-21 23:23:21,835 INFO [StoreOpener-da9d0eefced1af8dfad2324a5d9e1bf7-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:21,835 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(987): Running coprocessor post-open hooks for 1449fc3d526b1c6ab2fd158edec23277 2018-06-21 23:23:21,835 INFO [StoreOpener-04d71925444c7c823938ba3d9ad0504d-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:21,835 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(987): Running coprocessor post-open hooks for a657e5e90b9697e4cf81bc4bbde44800 2018-06-21 23:23:21,836 INFO [StoreOpener-da9d0eefced1af8dfad2324a5d9e1bf7-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:21,836 INFO [StoreOpener-04d71925444c7c823938ba3d9ad0504d-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:21,836 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(923): replaying wal for da9d0eefced1af8dfad2324a5d9e1bf7 2018-06-21 23:23:21,837 DEBUG [RS_OPEN_REGION-master/asf911:0-1] regionserver.HRegion(923): replaying wal for 04d71925444c7c823938ba3d9ad0504d 2018-06-21 23:23:21,837 DEBUG [RS_OPEN_REGION-master/asf911:0-1] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/04d71925444c7c823938ba3d9ad0504d 2018-06-21 23:23:21,838 DEBUG [RS_OPEN_REGION-master/asf911:0-1] regionserver.HRegion(931): stopping wal replay for 04d71925444c7c823938ba3d9ad0504d 2018-06-21 23:23:21,837 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/da9d0eefced1af8dfad2324a5d9e1bf7 2018-06-21 23:23:21,838 INFO [PostOpenDeployTasks:a657e5e90b9697e4cf81bc4bbde44800] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,www,1529623399386.a657e5e90b9697e4cf81bc4bbde44800. 2018-06-21 23:23:21,838 INFO [PostOpenDeployTasks:1449fc3d526b1c6ab2fd158edec23277] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,ooo,1529623399386.1449fc3d526b1c6ab2fd158edec23277. 2018-06-21 23:23:21,838 DEBUG [RS_OPEN_REGION-master/asf911:0-1] regionserver.HRegion(943): Cleaning up temporary data for 04d71925444c7c823938ba3d9ad0504d 2018-06-21 23:23:21,838 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(931): stopping wal replay for da9d0eefced1af8dfad2324a5d9e1bf7 2018-06-21 23:23:21,838 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(943): Cleaning up temporary data for da9d0eefced1af8dfad2324a5d9e1bf7 2018-06-21 23:23:21,839 DEBUG [RS_OPEN_REGION-master/asf911:0-1] regionserver.HRegion(954): Cleaning up detritus for 04d71925444c7c823938ba3d9ad0504d 2018-06-21 23:23:21,839 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(954): Cleaning up detritus for da9d0eefced1af8dfad2324a5d9e1bf7 2018-06-21 23:23:21,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=2, pid=22, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=1449fc3d526b1c6ab2fd158edec23277, target=asf911.gq1.ygridcore.net,56613,1529623390297; rit=OPENING, location=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:21,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=2, pid=30, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=a657e5e90b9697e4cf81bc4bbde44800, target=asf911.gq1.ygridcore.net,56613,1529623390297; rit=OPENING, location=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:21,842 DEBUG [PostOpenDeployTasks:1449fc3d526b1c6ab2fd158edec23277] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,ooo,1529623399386.1449fc3d526b1c6ab2fd158edec23277. 2018-06-21 23:23:21,843 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] handler.OpenRegionHandler(128): Opened multi_test_table,ooo,1529623399386.1449fc3d526b1c6ab2fd158edec23277. on asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:21,844 DEBUG [PostOpenDeployTasks:a657e5e90b9697e4cf81bc4bbde44800] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,www,1529623399386.a657e5e90b9697e4cf81bc4bbde44800. 2018-06-21 23:23:21,845 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] handler.OpenRegionHandler(128): Opened multi_test_table,www,1529623399386.a657e5e90b9697e4cf81bc4bbde44800. on asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:21,846 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(976): writing seq id for da9d0eefced1af8dfad2324a5d9e1bf7 2018-06-21 23:23:21,846 DEBUG [RS_OPEN_REGION-master/asf911:0-1] regionserver.HRegion(976): writing seq id for 04d71925444c7c823938ba3d9ad0504d 2018-06-21 23:23:21,850 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/da9d0eefced1af8dfad2324a5d9e1bf7/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-06-21 23:23:21,850 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(980): Opened da9d0eefced1af8dfad2324a5d9e1bf7; next sequenceid=2 2018-06-21 23:23:21,850 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(987): Running coprocessor post-open hooks for da9d0eefced1af8dfad2324a5d9e1bf7 2018-06-21 23:23:21,851 DEBUG [RS_OPEN_REGION-master/asf911:0-1] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/04d71925444c7c823938ba3d9ad0504d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-06-21 23:23:21,851 INFO [RS_OPEN_REGION-master/asf911:0-1] regionserver.HRegion(980): Opened 04d71925444c7c823938ba3d9ad0504d; next sequenceid=2 2018-06-21 23:23:21,851 DEBUG [RS_OPEN_REGION-master/asf911:0-1] regionserver.HRegion(987): Running coprocessor post-open hooks for 04d71925444c7c823938ba3d9ad0504d 2018-06-21 23:23:21,852 INFO [PostOpenDeployTasks:da9d0eefced1af8dfad2324a5d9e1bf7] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,fff,1529623399386.da9d0eefced1af8dfad2324a5d9e1bf7. 2018-06-21 23:23:21,853 INFO [PostOpenDeployTasks:04d71925444c7c823938ba3d9ad0504d] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,iii,1529623399386.04d71925444c7c823938ba3d9ad0504d. 2018-06-21 23:23:21,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=2, pid=13, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=da9d0eefced1af8dfad2324a5d9e1bf7, target=asf911.gq1.ygridcore.net,56613,1529623390297; rit=OPENING, location=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:21,856 DEBUG [PostOpenDeployTasks:04d71925444c7c823938ba3d9ad0504d] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=2, pid=16, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=04d71925444c7c823938ba3d9ad0504d, target=asf911.gq1.ygridcore.net,55146,1529623388534; rit=OPENING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:21,857 DEBUG [PostOpenDeployTasks:04d71925444c7c823938ba3d9ad0504d] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,iii,1529623399386.04d71925444c7c823938ba3d9ad0504d. 2018-06-21 23:23:21,858 DEBUG [RS_OPEN_REGION-master/asf911:0-1] handler.OpenRegionHandler(128): Opened multi_test_table,iii,1529623399386.04d71925444c7c823938ba3d9ad0504d. on asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:21,858 DEBUG [PostOpenDeployTasks:da9d0eefced1af8dfad2324a5d9e1bf7] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,fff,1529623399386.da9d0eefced1af8dfad2324a5d9e1bf7. 2018-06-21 23:23:21,859 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] handler.OpenRegionHandler(128): Opened multi_test_table,fff,1529623399386.da9d0eefced1af8dfad2324a5d9e1bf7. on asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:21,859 INFO [PEWorker-15] procedure2.ProcedureExecutor(1266): Finished pid=33, ppid=7, state=SUCCESS; AssignProcedure table=multi_test_table, region=794028a2b8fc0a516890763306629d4c, target=asf911.gq1.ygridcore.net,59970,1529623390066 in 663msec 2018-06-21 23:23:21,859 DEBUG [PEWorker-9] assignment.RegionTransitionProcedure(354): Finishing pid=16, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=04d71925444c7c823938ba3d9ad0504d, target=asf911.gq1.ygridcore.net,55146,1529623388534; rit=OPENING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:21,860 INFO [PEWorker-6] procedure2.ProcedureExecutor(1266): Finished pid=9, ppid=7, state=SUCCESS; AssignProcedure table=multi_test_table, region=e04a2e293b4c1bbd21e87d569c11154e, target=asf911.gq1.ygridcore.net,58235,1529623390382 in 668msec 2018-06-21 23:23:21,859 DEBUG [PEWorker-14] assignment.RegionTransitionProcedure(354): Finishing pid=13, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=da9d0eefced1af8dfad2324a5d9e1bf7, target=asf911.gq1.ygridcore.net,56613,1529623390297; rit=OPENING, location=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:21,860 INFO [PEWorker-9] assignment.RegionStateStore(198): pid=16 updating hbase:meta row=04d71925444c7c823938ba3d9ad0504d, regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:21,860 DEBUG [PEWorker-6] assignment.RegionTransitionProcedure(354): Finishing pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=cb0df54f72fae3cd39a03514dbbaa19c, target=asf911.gq1.ygridcore.net,58235,1529623390382; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:21,859 INFO [PEWorker-13] procedure2.ProcedureExecutor(1266): Finished pid=24, ppid=7, state=SUCCESS; AssignProcedure table=multi_test_table, region=7480a111a1f600aa34b97a77e47640bc, target=asf911.gq1.ygridcore.net,58235,1529623390382 in 668msec 2018-06-21 23:23:21,862 INFO [PEWorker-10] procedure2.ProcedureExecutor(1266): Finished pid=32, ppid=7, state=SUCCESS; AssignProcedure table=multi_test_table, region=7006d69958189f4e3b610c07f795d5d5, target=asf911.gq1.ygridcore.net,58235,1529623390382 in 689msec 2018-06-21 23:23:21,862 DEBUG [PEWorker-2] assignment.RegionTransitionProcedure(354): Finishing pid=22, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=1449fc3d526b1c6ab2fd158edec23277, target=asf911.gq1.ygridcore.net,56613,1529623390297; rit=OPENING, location=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:21,862 DEBUG [PEWorker-7] assignment.RegionTransitionProcedure(354): Finishing pid=17, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=573e96916daa12dfc083e39a253dc3a2, target=asf911.gq1.ygridcore.net,59970,1529623390066; rit=OPENING, location=asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,862 INFO [PEWorker-12] procedure2.ProcedureExecutor(1266): Finished pid=20, ppid=7, state=SUCCESS; AssignProcedure table=multi_test_table, region=28b5f974b61112896f4c4f1127dc2b99, target=asf911.gq1.ygridcore.net,59970,1529623390066 in 694msec 2018-06-21 23:23:21,860 DEBUG [PEWorker-5] assignment.RegionTransitionProcedure(354): Finishing pid=29, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=0dfb3ebb8c759d632ec793ac0387ec95, target=asf911.gq1.ygridcore.net,59970,1529623390066; rit=OPENING, location=asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,862 INFO [PEWorker-14] assignment.RegionStateStore(198): pid=13 updating hbase:meta row=da9d0eefced1af8dfad2324a5d9e1bf7, regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:21,860 DEBUG [PEWorker-4] assignment.RegionTransitionProcedure(354): Finishing pid=18, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=fee67ae5d4f2b2be35fa516877a8d173, target=asf911.gq1.ygridcore.net,55146,1529623388534; rit=OPENING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:21,860 DEBUG [PEWorker-3] assignment.RegionTransitionProcedure(354): Finishing pid=31, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=77a32d153e5d8148ca9d599c6a3cb695, target=asf911.gq1.ygridcore.net,55146,1529623388534; rit=OPENING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:21,860 INFO [PEWorker-11] procedure2.ProcedureExecutor(1266): Finished pid=25, ppid=7, state=SUCCESS; AssignProcedure table=multi_test_table, region=8e2537011449fca6295a357cbde76e6c, target=asf911.gq1.ygridcore.net,59970,1529623390066 in 686msec 2018-06-21 23:23:21,863 INFO [PEWorker-4] assignment.RegionStateStore(198): pid=18 updating hbase:meta row=fee67ae5d4f2b2be35fa516877a8d173, regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:21,860 DEBUG [PEWorker-15] assignment.RegionTransitionProcedure(354): Finishing pid=14, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=10cdebbb720d98123807841a0febf405, target=asf911.gq1.ygridcore.net,56613,1529623390297; rit=OPENING, location=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:21,863 INFO [PEWorker-3] assignment.RegionStateStore(198): pid=31 updating hbase:meta row=77a32d153e5d8148ca9d599c6a3cb695, regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:21,863 INFO [PEWorker-15] assignment.RegionStateStore(198): pid=14 updating hbase:meta row=10cdebbb720d98123807841a0febf405, regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:21,862 INFO [PEWorker-5] assignment.RegionStateStore(198): pid=29 updating hbase:meta row=0dfb3ebb8c759d632ec793ac0387ec95, regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,862 INFO [PEWorker-7] assignment.RegionStateStore(198): pid=17 updating hbase:meta row=573e96916daa12dfc083e39a253dc3a2, regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:21,862 INFO [PEWorker-2] assignment.RegionStateStore(198): pid=22 updating hbase:meta row=1449fc3d526b1c6ab2fd158edec23277, regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:21,862 INFO [RpcServer.priority.FPBQ.Fifo.handler=1,queue=0,port=40767] regionserver.RSRpcServices(1980): Open multi_test_table,eee,1529623399386.c793c2c3948aad84865d30ebf71a529e. 2018-06-21 23:23:21,862 INFO [PEWorker-16] assignment.RegionStateStore(198): pid=28 updating hbase:meta row=40503e5e9a757a4ccf579bf29b8d78c6, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:21,862 INFO [PEWorker-6] assignment.RegionStateStore(198): pid=8 updating hbase:meta row=cb0df54f72fae3cd39a03514dbbaa19c, regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:21,862 DEBUG [PEWorker-1] assignment.RegionTransitionProcedure(354): Finishing pid=30, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=a657e5e90b9697e4cf81bc4bbde44800, target=asf911.gq1.ygridcore.net,56613,1529623390297; rit=OPENING, location=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:21,862 DEBUG [PEWorker-8] assignment.RegionTransitionProcedure(354): Finishing pid=21, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=b1a8d69ffdd43e80f30a919effdc7c4e, target=asf911.gq1.ygridcore.net,55146,1529623388534; rit=OPENING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:21,865 INFO [PEWorker-1] assignment.RegionStateStore(198): pid=30 updating hbase:meta row=a657e5e90b9697e4cf81bc4bbde44800, regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:21,865 INFO [PEWorker-8] assignment.RegionStateStore(198): pid=21 updating hbase:meta row=b1a8d69ffdd43e80f30a919effdc7c4e, regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:21,871 INFO [RpcServer.priority.FPBQ.Fifo.handler=1,queue=0,port=40767] regionserver.RSRpcServices(1980): Open multi_test_table,ppp,1529623399386.36d0a953419e9f81636aebf73e7ed5df. 2018-06-21 23:23:21,871 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7094): Opening region: {ENCODED => c793c2c3948aad84865d30ebf71a529e, NAME => 'multi_test_table,eee,1529623399386.c793c2c3948aad84865d30ebf71a529e.', STARTKEY => 'eee', ENDKEY => 'fff'} 2018-06-21 23:23:21,871 INFO [RpcServer.priority.FPBQ.Fifo.handler=1,queue=0,port=40767] regionserver.RSRpcServices(1980): Open multi_test_table,sss,1529623399386.d4ac3b44d02d68507d6317e201672011. 2018-06-21 23:23:21,872 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7094): Opening region: {ENCODED => 36d0a953419e9f81636aebf73e7ed5df, NAME => 'multi_test_table,ppp,1529623399386.36d0a953419e9f81636aebf73e7ed5df.', STARTKEY => 'ppp', ENDKEY => 'qqq'} 2018-06-21 23:23:21,872 INFO [RpcServer.priority.FPBQ.Fifo.handler=1,queue=0,port=40767] regionserver.RSRpcServices(1980): Open multi_test_table,hhh,1529623399386.91ab16f39c84b448bdc2e5a1e1fc801b. 2018-06-21 23:23:21,872 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table c793c2c3948aad84865d30ebf71a529e 2018-06-21 23:23:21,872 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7094): Opening region: {ENCODED => d4ac3b44d02d68507d6317e201672011, NAME => 'multi_test_table,sss,1529623399386.d4ac3b44d02d68507d6317e201672011.', STARTKEY => 'sss', ENDKEY => 'ttt'} 2018-06-21 23:23:21,872 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table 36d0a953419e9f81636aebf73e7ed5df 2018-06-21 23:23:21,872 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(827): Instantiated multi_test_table,ppp,1529623399386.36d0a953419e9f81636aebf73e7ed5df.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:21,872 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(827): Instantiated multi_test_table,eee,1529623399386.c793c2c3948aad84865d30ebf71a529e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:21,873 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table d4ac3b44d02d68507d6317e201672011 2018-06-21 23:23:21,873 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7134): checking encryption for 36d0a953419e9f81636aebf73e7ed5df 2018-06-21 23:23:21,873 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7139): checking classloading for 36d0a953419e9f81636aebf73e7ed5df 2018-06-21 23:23:21,873 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(827): Instantiated multi_test_table,sss,1529623399386.d4ac3b44d02d68507d6317e201672011.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:21,873 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7134): checking encryption for c793c2c3948aad84865d30ebf71a529e 2018-06-21 23:23:21,873 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7139): checking classloading for c793c2c3948aad84865d30ebf71a529e 2018-06-21 23:23:21,873 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7134): checking encryption for d4ac3b44d02d68507d6317e201672011 2018-06-21 23:23:21,874 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7139): checking classloading for d4ac3b44d02d68507d6317e201672011 2018-06-21 23:23:21,883 DEBUG [StoreOpener-36d0a953419e9f81636aebf73e7ed5df-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/36d0a953419e9f81636aebf73e7ed5df/family 2018-06-21 23:23:21,883 DEBUG [StoreOpener-36d0a953419e9f81636aebf73e7ed5df-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/36d0a953419e9f81636aebf73e7ed5df/family 2018-06-21 23:23:21,883 DEBUG [StoreOpener-c793c2c3948aad84865d30ebf71a529e-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/c793c2c3948aad84865d30ebf71a529e/family 2018-06-21 23:23:21,883 DEBUG [RSProcedureDispatcher-pool3-t9] master.ServerManager(746): New admin connection to asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:21,883 DEBUG [StoreOpener-d4ac3b44d02d68507d6317e201672011-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/d4ac3b44d02d68507d6317e201672011/family 2018-06-21 23:23:21,884 DEBUG [StoreOpener-d4ac3b44d02d68507d6317e201672011-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/d4ac3b44d02d68507d6317e201672011/family 2018-06-21 23:23:21,883 DEBUG [StoreOpener-c793c2c3948aad84865d30ebf71a529e-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/c793c2c3948aad84865d30ebf71a529e/family 2018-06-21 23:23:21,884 INFO [StoreOpener-36d0a953419e9f81636aebf73e7ed5df-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:21,885 INFO [StoreOpener-36d0a953419e9f81636aebf73e7ed5df-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:21,888 INFO [StoreOpener-d4ac3b44d02d68507d6317e201672011-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:21,888 INFO [StoreOpener-c793c2c3948aad84865d30ebf71a529e-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:21,888 INFO [StoreOpener-36d0a953419e9f81636aebf73e7ed5df-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:21,888 INFO [StoreOpener-d4ac3b44d02d68507d6317e201672011-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:21,888 INFO [StoreOpener-c793c2c3948aad84865d30ebf71a529e-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:21,888 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(923): replaying wal for 36d0a953419e9f81636aebf73e7ed5df 2018-06-21 23:23:21,889 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-06-21 23:23:21,889 INFO [StoreOpener-d4ac3b44d02d68507d6317e201672011-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:21,890 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(923): replaying wal for d4ac3b44d02d68507d6317e201672011 2018-06-21 23:23:21,890 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/36d0a953419e9f81636aebf73e7ed5df 2018-06-21 23:23:21,890 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(931): stopping wal replay for 36d0a953419e9f81636aebf73e7ed5df 2018-06-21 23:23:21,890 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(943): Cleaning up temporary data for 36d0a953419e9f81636aebf73e7ed5df 2018-06-21 23:23:21,890 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 59309 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:21,890 INFO [StoreOpener-c793c2c3948aad84865d30ebf71a529e-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:21,891 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/d4ac3b44d02d68507d6317e201672011 2018-06-21 23:23:21,891 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(923): replaying wal for c793c2c3948aad84865d30ebf71a529e 2018-06-21 23:23:21,891 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(931): stopping wal replay for d4ac3b44d02d68507d6317e201672011 2018-06-21 23:23:21,892 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(943): Cleaning up temporary data for d4ac3b44d02d68507d6317e201672011 2018-06-21 23:23:21,892 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=52507] regionserver.RSRpcServices(1980): Open multi_test_table,ttt,1529623399386.dff6ab063ea78552b5e3c3ae7760ca5a. 2018-06-21 23:23:21,892 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(954): Cleaning up detritus for 36d0a953419e9f81636aebf73e7ed5df 2018-06-21 23:23:21,895 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/c793c2c3948aad84865d30ebf71a529e 2018-06-21 23:23:21,896 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(931): stopping wal replay for c793c2c3948aad84865d30ebf71a529e 2018-06-21 23:23:21,896 INFO [PEWorker-16] assignment.RegionTransitionProcedure(241): Dispatch pid=28, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=40503e5e9a757a4ccf579bf29b8d78c6, target=asf911.gq1.ygridcore.net,40767,1529623390148; rit=OPENING, location=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:21,896 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(954): Cleaning up detritus for d4ac3b44d02d68507d6317e201672011 2018-06-21 23:23:21,896 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(943): Cleaning up temporary data for c793c2c3948aad84865d30ebf71a529e 2018-06-21 23:23:21,897 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(954): Cleaning up detritus for c793c2c3948aad84865d30ebf71a529e 2018-06-21 23:23:21,898 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(976): writing seq id for 36d0a953419e9f81636aebf73e7ed5df 2018-06-21 23:23:21,898 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(976): writing seq id for d4ac3b44d02d68507d6317e201672011 2018-06-21 23:23:21,901 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=52507] regionserver.RSRpcServices(1980): Open multi_test_table,ddd,1529623399386.e66f718b26d1c4420497ccfb42f08321. 2018-06-21 23:23:21,901 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7094): Opening region: {ENCODED => dff6ab063ea78552b5e3c3ae7760ca5a, NAME => 'multi_test_table,ttt,1529623399386.dff6ab063ea78552b5e3c3ae7760ca5a.', STARTKEY => 'ttt', ENDKEY => 'uuu'} 2018-06-21 23:23:21,901 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=52507] regionserver.RSRpcServices(1980): Open multi_test_table,lll,1529623399386.77885b472341ce945c76aa06a4627146. 2018-06-21 23:23:21,901 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(976): writing seq id for c793c2c3948aad84865d30ebf71a529e 2018-06-21 23:23:21,901 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7094): Opening region: {ENCODED => e66f718b26d1c4420497ccfb42f08321, NAME => 'multi_test_table,ddd,1529623399386.e66f718b26d1c4420497ccfb42f08321.', STARTKEY => 'ddd', ENDKEY => 'eee'} 2018-06-21 23:23:21,902 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table dff6ab063ea78552b5e3c3ae7760ca5a 2018-06-21 23:23:21,902 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table e66f718b26d1c4420497ccfb42f08321 2018-06-21 23:23:21,903 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(827): Instantiated multi_test_table,ddd,1529623399386.e66f718b26d1c4420497ccfb42f08321.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:21,903 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7094): Opening region: {ENCODED => 77885b472341ce945c76aa06a4627146, NAME => 'multi_test_table,lll,1529623399386.77885b472341ce945c76aa06a4627146.', STARTKEY => 'lll', ENDKEY => 'mmm'} 2018-06-21 23:23:21,902 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(827): Instantiated multi_test_table,ttt,1529623399386.dff6ab063ea78552b5e3c3ae7760ca5a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:21,902 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=52507] regionserver.RSRpcServices(1980): Open multi_test_table,ccc,1529623399386.124da0ae361b264a91ea989bcd12b459. 2018-06-21 23:23:21,903 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table 77885b472341ce945c76aa06a4627146 2018-06-21 23:23:21,903 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7134): checking encryption for dff6ab063ea78552b5e3c3ae7760ca5a 2018-06-21 23:23:21,903 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7139): checking classloading for dff6ab063ea78552b5e3c3ae7760ca5a 2018-06-21 23:23:21,903 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7134): checking encryption for e66f718b26d1c4420497ccfb42f08321 2018-06-21 23:23:21,903 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(827): Instantiated multi_test_table,lll,1529623399386.77885b472341ce945c76aa06a4627146.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:21,904 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7139): checking classloading for e66f718b26d1c4420497ccfb42f08321 2018-06-21 23:23:21,904 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7134): checking encryption for 77885b472341ce945c76aa06a4627146 2018-06-21 23:23:21,904 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7139): checking classloading for 77885b472341ce945c76aa06a4627146 2018-06-21 23:23:21,905 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/d4ac3b44d02d68507d6317e201672011/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-06-21 23:23:21,905 INFO [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(980): Opened d4ac3b44d02d68507d6317e201672011; next sequenceid=2 2018-06-21 23:23:21,905 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(987): Running coprocessor post-open hooks for d4ac3b44d02d68507d6317e201672011 2018-06-21 23:23:21,907 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/36d0a953419e9f81636aebf73e7ed5df/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-06-21 23:23:21,907 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(980): Opened 36d0a953419e9f81636aebf73e7ed5df; next sequenceid=2 2018-06-21 23:23:21,907 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(987): Running coprocessor post-open hooks for 36d0a953419e9f81636aebf73e7ed5df 2018-06-21 23:23:21,933 INFO [PostOpenDeployTasks:36d0a953419e9f81636aebf73e7ed5df] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,ppp,1529623399386.36d0a953419e9f81636aebf73e7ed5df. 2018-06-21 23:23:21,933 INFO [PostOpenDeployTasks:d4ac3b44d02d68507d6317e201672011] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,sss,1529623399386.d4ac3b44d02d68507d6317e201672011. 2018-06-21 23:23:21,934 DEBUG [StoreOpener-e66f718b26d1c4420497ccfb42f08321-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/e66f718b26d1c4420497ccfb42f08321/family 2018-06-21 23:23:21,934 DEBUG [StoreOpener-e66f718b26d1c4420497ccfb42f08321-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/e66f718b26d1c4420497ccfb42f08321/family 2018-06-21 23:23:21,934 DEBUG [StoreOpener-dff6ab063ea78552b5e3c3ae7760ca5a-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/dff6ab063ea78552b5e3c3ae7760ca5a/family 2018-06-21 23:23:21,934 DEBUG [StoreOpener-dff6ab063ea78552b5e3c3ae7760ca5a-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/dff6ab063ea78552b5e3c3ae7760ca5a/family 2018-06-21 23:23:21,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=2, pid=23, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=36d0a953419e9f81636aebf73e7ed5df, target=asf911.gq1.ygridcore.net,40767,1529623390148; rit=OPENING, location=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:21,939 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/c793c2c3948aad84865d30ebf71a529e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-06-21 23:23:21,934 DEBUG [StoreOpener-77885b472341ce945c76aa06a4627146-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/77885b472341ce945c76aa06a4627146/family 2018-06-21 23:23:21,940 DEBUG [StoreOpener-77885b472341ce945c76aa06a4627146-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/77885b472341ce945c76aa06a4627146/family 2018-06-21 23:23:21,940 DEBUG [PEWorker-13] assignment.RegionTransitionProcedure(354): Finishing pid=23, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=36d0a953419e9f81636aebf73e7ed5df, target=asf911.gq1.ygridcore.net,40767,1529623390148; rit=OPENING, location=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:21,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=2, pid=26, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=d4ac3b44d02d68507d6317e201672011, target=asf911.gq1.ygridcore.net,40767,1529623390148; rit=OPENING, location=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:21,940 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(980): Opened c793c2c3948aad84865d30ebf71a529e; next sequenceid=2 2018-06-21 23:23:21,942 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(987): Running coprocessor post-open hooks for c793c2c3948aad84865d30ebf71a529e 2018-06-21 23:23:21,942 INFO [StoreOpener-e66f718b26d1c4420497ccfb42f08321-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:21,942 INFO [PEWorker-13] assignment.RegionStateStore(198): pid=23 updating hbase:meta row=36d0a953419e9f81636aebf73e7ed5df, regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:21,942 INFO [StoreOpener-dff6ab063ea78552b5e3c3ae7760ca5a-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:21,941 DEBUG [PostOpenDeployTasks:36d0a953419e9f81636aebf73e7ed5df] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,ppp,1529623399386.36d0a953419e9f81636aebf73e7ed5df. 2018-06-21 23:23:21,942 DEBUG [PostOpenDeployTasks:d4ac3b44d02d68507d6317e201672011] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,sss,1529623399386.d4ac3b44d02d68507d6317e201672011. 2018-06-21 23:23:21,942 DEBUG [PEWorker-10] assignment.RegionTransitionProcedure(354): Finishing pid=26, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=d4ac3b44d02d68507d6317e201672011, target=asf911.gq1.ygridcore.net,40767,1529623390148; rit=OPENING, location=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:21,942 INFO [StoreOpener-77885b472341ce945c76aa06a4627146-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:21,945 INFO [PEWorker-10] assignment.RegionStateStore(198): pid=26 updating hbase:meta row=d4ac3b44d02d68507d6317e201672011, regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:21,942 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] handler.OpenRegionHandler(128): Opened multi_test_table,ppp,1529623399386.36d0a953419e9f81636aebf73e7ed5df. on asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:21,942 INFO [StoreOpener-e66f718b26d1c4420497ccfb42f08321-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:21,945 INFO [StoreOpener-dff6ab063ea78552b5e3c3ae7760ca5a-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:21,945 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] handler.OpenRegionHandler(128): Opened multi_test_table,sss,1529623399386.d4ac3b44d02d68507d6317e201672011. on asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:21,946 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7094): Opening region: {ENCODED => 91ab16f39c84b448bdc2e5a1e1fc801b, NAME => 'multi_test_table,hhh,1529623399386.91ab16f39c84b448bdc2e5a1e1fc801b.', STARTKEY => 'hhh', ENDKEY => 'iii'} 2018-06-21 23:23:21,946 INFO [StoreOpener-77885b472341ce945c76aa06a4627146-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:21,946 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table 91ab16f39c84b448bdc2e5a1e1fc801b 2018-06-21 23:23:21,947 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(827): Instantiated multi_test_table,hhh,1529623399386.91ab16f39c84b448bdc2e5a1e1fc801b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:21,947 INFO [PostOpenDeployTasks:c793c2c3948aad84865d30ebf71a529e] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,eee,1529623399386.c793c2c3948aad84865d30ebf71a529e. 2018-06-21 23:23:21,947 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7134): checking encryption for 91ab16f39c84b448bdc2e5a1e1fc801b 2018-06-21 23:23:21,947 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7139): checking classloading for 91ab16f39c84b448bdc2e5a1e1fc801b 2018-06-21 23:23:21,948 INFO [StoreOpener-77885b472341ce945c76aa06a4627146-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:21,948 INFO [StoreOpener-e66f718b26d1c4420497ccfb42f08321-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:21,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=2, pid=12, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=c793c2c3948aad84865d30ebf71a529e, target=asf911.gq1.ygridcore.net,40767,1529623390148; rit=OPENING, location=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:21,948 INFO [StoreOpener-dff6ab063ea78552b5e3c3ae7760ca5a-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:21,948 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(923): replaying wal for e66f718b26d1c4420497ccfb42f08321 2018-06-21 23:23:21,948 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(923): replaying wal for 77885b472341ce945c76aa06a4627146 2018-06-21 23:23:21,948 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(923): replaying wal for dff6ab063ea78552b5e3c3ae7760ca5a 2018-06-21 23:23:21,949 DEBUG [PEWorker-12] assignment.RegionTransitionProcedure(354): Finishing pid=12, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=c793c2c3948aad84865d30ebf71a529e, target=asf911.gq1.ygridcore.net,40767,1529623390148; rit=OPENING, location=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:21,949 DEBUG [PostOpenDeployTasks:c793c2c3948aad84865d30ebf71a529e] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,eee,1529623399386.c793c2c3948aad84865d30ebf71a529e. 2018-06-21 23:23:21,950 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/dff6ab063ea78552b5e3c3ae7760ca5a 2018-06-21 23:23:21,950 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/e66f718b26d1c4420497ccfb42f08321 2018-06-21 23:23:21,953 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(931): stopping wal replay for e66f718b26d1c4420497ccfb42f08321 2018-06-21 23:23:21,953 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/77885b472341ce945c76aa06a4627146 2018-06-21 23:23:21,953 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(931): stopping wal replay for 77885b472341ce945c76aa06a4627146 2018-06-21 23:23:21,953 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(931): stopping wal replay for dff6ab063ea78552b5e3c3ae7760ca5a 2018-06-21 23:23:21,954 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(943): Cleaning up temporary data for dff6ab063ea78552b5e3c3ae7760ca5a 2018-06-21 23:23:21,953 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] handler.OpenRegionHandler(128): Opened multi_test_table,eee,1529623399386.c793c2c3948aad84865d30ebf71a529e. on asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:21,952 INFO [PEWorker-12] assignment.RegionStateStore(198): pid=12 updating hbase:meta row=c793c2c3948aad84865d30ebf71a529e, regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:21,954 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(943): Cleaning up temporary data for 77885b472341ce945c76aa06a4627146 2018-06-21 23:23:21,953 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(943): Cleaning up temporary data for e66f718b26d1c4420497ccfb42f08321 2018-06-21 23:23:21,955 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(954): Cleaning up detritus for dff6ab063ea78552b5e3c3ae7760ca5a 2018-06-21 23:23:21,955 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(954): Cleaning up detritus for 77885b472341ce945c76aa06a4627146 2018-06-21 23:23:21,956 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(954): Cleaning up detritus for e66f718b26d1c4420497ccfb42f08321 2018-06-21 23:23:21,957 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(976): writing seq id for dff6ab063ea78552b5e3c3ae7760ca5a 2018-06-21 23:23:21,957 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(976): writing seq id for 77885b472341ce945c76aa06a4627146 2018-06-21 23:23:21,958 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(976): writing seq id for e66f718b26d1c4420497ccfb42f08321 2018-06-21 23:23:21,988 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/e66f718b26d1c4420497ccfb42f08321/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-06-21 23:23:21,988 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/77885b472341ce945c76aa06a4627146/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-06-21 23:23:21,988 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(980): Opened e66f718b26d1c4420497ccfb42f08321; next sequenceid=2 2018-06-21 23:23:21,989 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(987): Running coprocessor post-open hooks for e66f718b26d1c4420497ccfb42f08321 2018-06-21 23:23:21,988 DEBUG [StoreOpener-91ab16f39c84b448bdc2e5a1e1fc801b-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/91ab16f39c84b448bdc2e5a1e1fc801b/family 2018-06-21 23:23:21,988 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/dff6ab063ea78552b5e3c3ae7760ca5a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-06-21 23:23:21,989 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(980): Opened dff6ab063ea78552b5e3c3ae7760ca5a; next sequenceid=2 2018-06-21 23:23:21,989 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(987): Running coprocessor post-open hooks for dff6ab063ea78552b5e3c3ae7760ca5a 2018-06-21 23:23:21,988 INFO [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(980): Opened 77885b472341ce945c76aa06a4627146; next sequenceid=2 2018-06-21 23:23:21,989 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(987): Running coprocessor post-open hooks for 77885b472341ce945c76aa06a4627146 2018-06-21 23:23:21,989 DEBUG [StoreOpener-91ab16f39c84b448bdc2e5a1e1fc801b-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/91ab16f39c84b448bdc2e5a1e1fc801b/family 2018-06-21 23:23:21,993 INFO [StoreOpener-91ab16f39c84b448bdc2e5a1e1fc801b-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:21,993 INFO [StoreOpener-91ab16f39c84b448bdc2e5a1e1fc801b-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:21,993 INFO [PostOpenDeployTasks:e66f718b26d1c4420497ccfb42f08321] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,ddd,1529623399386.e66f718b26d1c4420497ccfb42f08321. 2018-06-21 23:23:21,994 INFO [PEWorker-4] procedure2.ProcedureExecutor(1266): Finished pid=18, ppid=7, state=SUCCESS; AssignProcedure table=multi_test_table, region=fee67ae5d4f2b2be35fa516877a8d173, target=asf911.gq1.ygridcore.net,55146,1529623388534 in 842msec 2018-06-21 23:23:21,998 INFO [PEWorker-5] procedure2.ProcedureExecutor(1266): Finished pid=29, ppid=7, state=SUCCESS; AssignProcedure table=multi_test_table, region=0dfb3ebb8c759d632ec793ac0387ec95, target=asf911.gq1.ygridcore.net,59970,1529623390066 in 837msec 2018-06-21 23:23:21,998 INFO [PEWorker-3] procedure2.ProcedureExecutor(1266): Finished pid=31, ppid=7, state=SUCCESS; AssignProcedure table=multi_test_table, region=77a32d153e5d8148ca9d599c6a3cb695, target=asf911.gq1.ygridcore.net,55146,1529623388534 in 837msec 2018-06-21 23:23:21,999 INFO [PEWorker-7] procedure2.ProcedureExecutor(1266): Finished pid=17, ppid=7, state=SUCCESS; AssignProcedure table=multi_test_table, region=573e96916daa12dfc083e39a253dc3a2, target=asf911.gq1.ygridcore.net,59970,1529623390066 in 850msec 2018-06-21 23:23:21,998 INFO [PostOpenDeployTasks:77885b472341ce945c76aa06a4627146] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,lll,1529623399386.77885b472341ce945c76aa06a4627146. 2018-06-21 23:23:21,997 INFO [PostOpenDeployTasks:dff6ab063ea78552b5e3c3ae7760ca5a] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,ttt,1529623399386.dff6ab063ea78552b5e3c3ae7760ca5a. 2018-06-21 23:23:21,999 INFO [StoreOpener-91ab16f39c84b448bdc2e5a1e1fc801b-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:21,994 INFO [PEWorker-9] procedure2.ProcedureExecutor(1266): Finished pid=16, ppid=7, state=SUCCESS; AssignProcedure table=multi_test_table, region=04d71925444c7c823938ba3d9ad0504d, target=asf911.gq1.ygridcore.net,55146,1529623388534 in 842msec 2018-06-21 23:23:21,998 INFO [PEWorker-8] procedure2.ProcedureExecutor(1266): Finished pid=21, ppid=7, state=SUCCESS; AssignProcedure table=multi_test_table, region=b1a8d69ffdd43e80f30a919effdc7c4e, target=asf911.gq1.ygridcore.net,55146,1529623388534 in 846msec 2018-06-21 23:23:21,998 INFO [PEWorker-1] procedure2.ProcedureExecutor(1266): Finished pid=30, ppid=7, state=SUCCESS; AssignProcedure table=multi_test_table, region=a657e5e90b9697e4cf81bc4bbde44800, target=asf911.gq1.ygridcore.net,56613,1529623390297 in 840msec 2018-06-21 23:23:21,998 INFO [PEWorker-2] procedure2.ProcedureExecutor(1266): Finished pid=22, ppid=7, state=SUCCESS; AssignProcedure table=multi_test_table, region=1449fc3d526b1c6ab2fd158edec23277, target=asf911.gq1.ygridcore.net,56613,1529623390297 in 845msec 2018-06-21 23:23:21,998 INFO [PEWorker-6] procedure2.ProcedureExecutor(1266): Finished pid=8, ppid=7, state=SUCCESS; AssignProcedure table=multi_test_table, region=cb0df54f72fae3cd39a03514dbbaa19c, target=asf911.gq1.ygridcore.net,58235,1529623390382 in 845msec 2018-06-21 23:23:21,998 INFO [PEWorker-14] procedure2.ProcedureExecutor(1266): Finished pid=13, ppid=7, state=SUCCESS; AssignProcedure table=multi_test_table, region=da9d0eefced1af8dfad2324a5d9e1bf7, target=asf911.gq1.ygridcore.net,56613,1529623390297 in 842msec 2018-06-21 23:23:21,998 INFO [PEWorker-15] procedure2.ProcedureExecutor(1266): Finished pid=14, ppid=7, state=SUCCESS; AssignProcedure table=multi_test_table, region=10cdebbb720d98123807841a0febf405, target=asf911.gq1.ygridcore.net,56613,1529623390297 in 842msec 2018-06-21 23:23:22,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=2, pid=19, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=77885b472341ce945c76aa06a4627146, target=asf911.gq1.ygridcore.net,52507,1529623390228; rit=OPENING, location=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:22,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=2, pid=11, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=e66f718b26d1c4420497ccfb42f08321, target=asf911.gq1.ygridcore.net,52507,1529623390228; rit=OPENING, location=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:21,999 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(923): replaying wal for 91ab16f39c84b448bdc2e5a1e1fc801b 2018-06-21 23:23:22,000 DEBUG [PEWorker-4] assignment.RegionTransitionProcedure(354): Finishing pid=11, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=e66f718b26d1c4420497ccfb42f08321, target=asf911.gq1.ygridcore.net,52507,1529623390228; rit=OPENING, location=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:22,000 DEBUG [PEWorker-11] assignment.RegionTransitionProcedure(354): Finishing pid=19, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=77885b472341ce945c76aa06a4627146, target=asf911.gq1.ygridcore.net,52507,1529623390228; rit=OPENING, location=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:22,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=2, pid=27, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=dff6ab063ea78552b5e3c3ae7760ca5a, target=asf911.gq1.ygridcore.net,52507,1529623390228; rit=OPENING, location=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:22,005 INFO [PEWorker-4] assignment.RegionStateStore(198): pid=11 updating hbase:meta row=e66f718b26d1c4420497ccfb42f08321, regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:22,003 DEBUG [PostOpenDeployTasks:77885b472341ce945c76aa06a4627146] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,lll,1529623399386.77885b472341ce945c76aa06a4627146. 2018-06-21 23:23:22,005 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/91ab16f39c84b448bdc2e5a1e1fc801b 2018-06-21 23:23:22,008 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(931): stopping wal replay for 91ab16f39c84b448bdc2e5a1e1fc801b 2018-06-21 23:23:22,004 DEBUG [PostOpenDeployTasks:e66f718b26d1c4420497ccfb42f08321] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,ddd,1529623399386.e66f718b26d1c4420497ccfb42f08321. 2018-06-21 23:23:22,003 INFO [PEWorker-11] assignment.RegionStateStore(198): pid=19 updating hbase:meta row=77885b472341ce945c76aa06a4627146, regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:22,008 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(943): Cleaning up temporary data for 91ab16f39c84b448bdc2e5a1e1fc801b 2018-06-21 23:23:22,010 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] handler.OpenRegionHandler(128): Opened multi_test_table,ddd,1529623399386.e66f718b26d1c4420497ccfb42f08321. on asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:22,008 DEBUG [PostOpenDeployTasks:dff6ab063ea78552b5e3c3ae7760ca5a] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,ttt,1529623399386.dff6ab063ea78552b5e3c3ae7760ca5a. 2018-06-21 23:23:22,008 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] handler.OpenRegionHandler(128): Opened multi_test_table,lll,1529623399386.77885b472341ce945c76aa06a4627146. on asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:22,005 DEBUG [PEWorker-16] assignment.RegionTransitionProcedure(354): Finishing pid=27, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=dff6ab063ea78552b5e3c3ae7760ca5a, target=asf911.gq1.ygridcore.net,52507,1529623390228; rit=OPENING, location=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:22,015 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(954): Cleaning up detritus for 91ab16f39c84b448bdc2e5a1e1fc801b 2018-06-21 23:23:22,014 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] handler.OpenRegionHandler(128): Opened multi_test_table,ttt,1529623399386.dff6ab063ea78552b5e3c3ae7760ca5a. on asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:22,014 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7094): Opening region: {ENCODED => 124da0ae361b264a91ea989bcd12b459, NAME => 'multi_test_table,ccc,1529623399386.124da0ae361b264a91ea989bcd12b459.', STARTKEY => 'ccc', ENDKEY => 'ddd'} 2018-06-21 23:23:22,018 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table 124da0ae361b264a91ea989bcd12b459 2018-06-21 23:23:22,019 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(827): Instantiated multi_test_table,ccc,1529623399386.124da0ae361b264a91ea989bcd12b459.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:22,019 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7134): checking encryption for 124da0ae361b264a91ea989bcd12b459 2018-06-21 23:23:22,019 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7139): checking classloading for 124da0ae361b264a91ea989bcd12b459 2018-06-21 23:23:22,019 INFO [PEWorker-16] assignment.RegionStateStore(198): pid=27 updating hbase:meta row=dff6ab063ea78552b5e3c3ae7760ca5a, regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:22,020 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(976): writing seq id for 91ab16f39c84b448bdc2e5a1e1fc801b 2018-06-21 23:23:22,034 DEBUG [StoreOpener-124da0ae361b264a91ea989bcd12b459-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/124da0ae361b264a91ea989bcd12b459/family 2018-06-21 23:23:22,034 DEBUG [StoreOpener-124da0ae361b264a91ea989bcd12b459-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/124da0ae361b264a91ea989bcd12b459/family 2018-06-21 23:23:22,035 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/91ab16f39c84b448bdc2e5a1e1fc801b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-06-21 23:23:22,035 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(980): Opened 91ab16f39c84b448bdc2e5a1e1fc801b; next sequenceid=2 2018-06-21 23:23:22,035 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(987): Running coprocessor post-open hooks for 91ab16f39c84b448bdc2e5a1e1fc801b 2018-06-21 23:23:22,035 INFO [StoreOpener-124da0ae361b264a91ea989bcd12b459-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:22,036 INFO [StoreOpener-124da0ae361b264a91ea989bcd12b459-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:22,037 INFO [StoreOpener-124da0ae361b264a91ea989bcd12b459-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:22,037 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(923): replaying wal for 124da0ae361b264a91ea989bcd12b459 2018-06-21 23:23:22,037 INFO [PostOpenDeployTasks:91ab16f39c84b448bdc2e5a1e1fc801b] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,hhh,1529623399386.91ab16f39c84b448bdc2e5a1e1fc801b. 2018-06-21 23:23:22,038 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/124da0ae361b264a91ea989bcd12b459 2018-06-21 23:23:22,038 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(931): stopping wal replay for 124da0ae361b264a91ea989bcd12b459 2018-06-21 23:23:22,038 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(943): Cleaning up temporary data for 124da0ae361b264a91ea989bcd12b459 2018-06-21 23:23:22,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=2, pid=15, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=91ab16f39c84b448bdc2e5a1e1fc801b, target=asf911.gq1.ygridcore.net,40767,1529623390148; rit=OPENING, location=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:22,039 DEBUG [PEWorker-5] assignment.RegionTransitionProcedure(354): Finishing pid=15, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=91ab16f39c84b448bdc2e5a1e1fc801b, target=asf911.gq1.ygridcore.net,40767,1529623390148; rit=OPENING, location=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:22,041 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(954): Cleaning up detritus for 124da0ae361b264a91ea989bcd12b459 2018-06-21 23:23:22,042 INFO [PEWorker-5] assignment.RegionStateStore(198): pid=15 updating hbase:meta row=91ab16f39c84b448bdc2e5a1e1fc801b, regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:22,042 DEBUG [PostOpenDeployTasks:91ab16f39c84b448bdc2e5a1e1fc801b] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,hhh,1529623399386.91ab16f39c84b448bdc2e5a1e1fc801b. 2018-06-21 23:23:22,042 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] handler.OpenRegionHandler(128): Opened multi_test_table,hhh,1529623399386.91ab16f39c84b448bdc2e5a1e1fc801b. on asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:22,044 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(976): writing seq id for 124da0ae361b264a91ea989bcd12b459 2018-06-21 23:23:22,047 INFO [RpcServer.priority.FPBQ.Fifo.handler=0,queue=0,port=40767] regionserver.RSRpcServices(1980): Open multi_test_table,uuu,1529623399386.40503e5e9a757a4ccf579bf29b8d78c6. 2018-06-21 23:23:22,048 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/124da0ae361b264a91ea989bcd12b459/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-06-21 23:23:22,048 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(980): Opened 124da0ae361b264a91ea989bcd12b459; next sequenceid=2 2018-06-21 23:23:22,049 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(987): Running coprocessor post-open hooks for 124da0ae361b264a91ea989bcd12b459 2018-06-21 23:23:22,053 INFO [PostOpenDeployTasks:124da0ae361b264a91ea989bcd12b459] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,ccc,1529623399386.124da0ae361b264a91ea989bcd12b459. 2018-06-21 23:23:22,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=2, pid=10, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=124da0ae361b264a91ea989bcd12b459, target=asf911.gq1.ygridcore.net,52507,1529623390228; rit=OPENING, location=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:22,054 DEBUG [PEWorker-3] assignment.RegionTransitionProcedure(354): Finishing pid=10, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=124da0ae361b264a91ea989bcd12b459, target=asf911.gq1.ygridcore.net,52507,1529623390228; rit=OPENING, location=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:22,054 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7094): Opening region: {ENCODED => 40503e5e9a757a4ccf579bf29b8d78c6, NAME => 'multi_test_table,uuu,1529623399386.40503e5e9a757a4ccf579bf29b8d78c6.', STARTKEY => 'uuu', ENDKEY => 'vvv'} 2018-06-21 23:23:22,057 INFO [PEWorker-3] assignment.RegionStateStore(198): pid=10 updating hbase:meta row=124da0ae361b264a91ea989bcd12b459, regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:22,058 DEBUG [PostOpenDeployTasks:124da0ae361b264a91ea989bcd12b459] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,ccc,1529623399386.124da0ae361b264a91ea989bcd12b459. 2018-06-21 23:23:22,058 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table 40503e5e9a757a4ccf579bf29b8d78c6 2018-06-21 23:23:22,058 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] handler.OpenRegionHandler(128): Opened multi_test_table,ccc,1529623399386.124da0ae361b264a91ea989bcd12b459. on asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:22,058 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(827): Instantiated multi_test_table,uuu,1529623399386.40503e5e9a757a4ccf579bf29b8d78c6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:22,058 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7134): checking encryption for 40503e5e9a757a4ccf579bf29b8d78c6 2018-06-21 23:23:22,058 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7139): checking classloading for 40503e5e9a757a4ccf579bf29b8d78c6 2018-06-21 23:23:22,068 DEBUG [StoreOpener-40503e5e9a757a4ccf579bf29b8d78c6-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/40503e5e9a757a4ccf579bf29b8d78c6/family 2018-06-21 23:23:22,068 DEBUG [StoreOpener-40503e5e9a757a4ccf579bf29b8d78c6-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/40503e5e9a757a4ccf579bf29b8d78c6/family 2018-06-21 23:23:22,069 INFO [StoreOpener-40503e5e9a757a4ccf579bf29b8d78c6-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:22,069 INFO [StoreOpener-40503e5e9a757a4ccf579bf29b8d78c6-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:22,070 INFO [StoreOpener-40503e5e9a757a4ccf579bf29b8d78c6-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:22,071 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(923): replaying wal for 40503e5e9a757a4ccf579bf29b8d78c6 2018-06-21 23:23:22,072 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/40503e5e9a757a4ccf579bf29b8d78c6 2018-06-21 23:23:22,072 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(931): stopping wal replay for 40503e5e9a757a4ccf579bf29b8d78c6 2018-06-21 23:23:22,072 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(943): Cleaning up temporary data for 40503e5e9a757a4ccf579bf29b8d78c6 2018-06-21 23:23:22,073 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(954): Cleaning up detritus for 40503e5e9a757a4ccf579bf29b8d78c6 2018-06-21 23:23:22,075 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(976): writing seq id for 40503e5e9a757a4ccf579bf29b8d78c6 2018-06-21 23:23:22,079 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/40503e5e9a757a4ccf579bf29b8d78c6/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-06-21 23:23:22,080 INFO [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(980): Opened 40503e5e9a757a4ccf579bf29b8d78c6; next sequenceid=2 2018-06-21 23:23:22,080 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(987): Running coprocessor post-open hooks for 40503e5e9a757a4ccf579bf29b8d78c6 2018-06-21 23:23:22,082 INFO [PostOpenDeployTasks:40503e5e9a757a4ccf579bf29b8d78c6] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,uuu,1529623399386.40503e5e9a757a4ccf579bf29b8d78c6. 2018-06-21 23:23:22,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=2, pid=28, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=40503e5e9a757a4ccf579bf29b8d78c6, target=asf911.gq1.ygridcore.net,40767,1529623390148; rit=OPENING, location=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:22,089 DEBUG [PEWorker-7] assignment.RegionTransitionProcedure(354): Finishing pid=28, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=40503e5e9a757a4ccf579bf29b8d78c6, target=asf911.gq1.ygridcore.net,40767,1529623390148; rit=OPENING, location=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:22,091 INFO [PEWorker-7] assignment.RegionStateStore(198): pid=28 updating hbase:meta row=40503e5e9a757a4ccf579bf29b8d78c6, regionState=OPEN, openSeqNum=2, regionLocation=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:22,091 DEBUG [PostOpenDeployTasks:40503e5e9a757a4ccf579bf29b8d78c6] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,uuu,1529623399386.40503e5e9a757a4ccf579bf29b8d78c6. 2018-06-21 23:23:22,091 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] handler.OpenRegionHandler(128): Opened multi_test_table,uuu,1529623399386.40503e5e9a757a4ccf579bf29b8d78c6. on asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:22,152 INFO [PEWorker-13] procedure2.ProcedureExecutor(1266): Finished pid=23, ppid=7, state=SUCCESS; AssignProcedure table=multi_test_table, region=36d0a953419e9f81636aebf73e7ed5df, target=asf911.gq1.ygridcore.net,40767,1529623390148 in 903msec 2018-06-21 23:23:22,152 INFO [PEWorker-12] procedure2.ProcedureExecutor(1266): Finished pid=12, ppid=7, state=SUCCESS; AssignProcedure table=multi_test_table, region=c793c2c3948aad84865d30ebf71a529e, target=asf911.gq1.ygridcore.net,40767,1529623390148 in 942msec 2018-06-21 23:23:22,152 INFO [PEWorker-3] procedure2.ProcedureExecutor(1266): Finished pid=10, ppid=7, state=SUCCESS; AssignProcedure table=multi_test_table, region=124da0ae361b264a91ea989bcd12b459, target=asf911.gq1.ygridcore.net,52507,1529623390228 in 1.0220sec 2018-06-21 23:23:22,152 INFO [PEWorker-5] procedure2.ProcedureExecutor(1266): Finished pid=15, ppid=7, state=SUCCESS; AssignProcedure table=multi_test_table, region=91ab16f39c84b448bdc2e5a1e1fc801b, target=asf911.gq1.ygridcore.net,40767,1529623390148 in 1.0000sec 2018-06-21 23:23:22,152 INFO [PEWorker-11] procedure2.ProcedureExecutor(1266): Finished pid=19, ppid=7, state=SUCCESS; AssignProcedure table=multi_test_table, region=77885b472341ce945c76aa06a4627146, target=asf911.gq1.ygridcore.net,52507,1529623390228 in 989msec 2018-06-21 23:23:22,152 INFO [PEWorker-4] procedure2.ProcedureExecutor(1266): Finished pid=11, ppid=7, state=SUCCESS; AssignProcedure table=multi_test_table, region=e66f718b26d1c4420497ccfb42f08321, target=asf911.gq1.ygridcore.net,52507,1529623390228 in 989msec 2018-06-21 23:23:22,152 INFO [PEWorker-16] procedure2.ProcedureExecutor(1266): Finished pid=27, ppid=7, state=SUCCESS; AssignProcedure table=multi_test_table, region=dff6ab063ea78552b5e3c3ae7760ca5a, target=asf911.gq1.ygridcore.net,52507,1529623390228 in 984msec 2018-06-21 23:23:22,152 INFO [PEWorker-10] procedure2.ProcedureExecutor(1266): Finished pid=26, ppid=7, state=SUCCESS; AssignProcedure table=multi_test_table, region=d4ac3b44d02d68507d6317e201672011, target=asf911.gq1.ygridcore.net,40767,1529623390148 in 901msec 2018-06-21 23:23:22,364 INFO [PEWorker-7] procedure2.ProcedureExecutor(1635): Finished subprocedure(s) of pid=7, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=multi_test_table; resume parent processing. 2018-06-21 23:23:22,364 INFO [PEWorker-7] procedure2.ProcedureExecutor(1266): Finished pid=28, ppid=7, state=SUCCESS; AssignProcedure table=multi_test_table, region=40503e5e9a757a4ccf579bf29b8d78c6, target=asf911.gq1.ygridcore.net,40767,1529623390148 in 1.0440sec 2018-06-21 23:23:22,364 DEBUG [PEWorker-9] hbase.MetaTableAccessor(2153): Put {"totalColumns":1,"row":"multi_test_table","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1529623402364}]},"ts":1529623402364} 2018-06-21 23:23:22,368 INFO [PEWorker-9] hbase.MetaTableAccessor(1673): Updated tableName=multi_test_table, state=ENABLED in hbase:meta 2018-06-21 23:23:22,548 INFO [PEWorker-9] procedure2.ProcedureExecutor(1266): Finished pid=7, state=SUCCESS; CreateTableProcedure table=multi_test_table in 3.0920sec 2018-06-21 23:23:23,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=55146] master.MasterRpcServices(1142): Checking to see if procedure is done pid=7 2018-06-21 23:23:23,820 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3668): Operation: CREATE, Table Name: default:multi_test_table completed 2018-06-21 23:23:23,820 DEBUG [Time-limited test] hbase.HBaseTestingUtility(3284): Waiting until all regions of table multi_test_table get assigned. Timeout = 60000ms 2018-06-21 23:23:23,824 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2018-06-21 23:23:23,862 INFO [Time-limited test] hbase.HBaseTestingUtility(3336): All regions for table multi_test_table assigned to meta. Checking AM states. 2018-06-21 23:23:23,863 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2018-06-21 23:23:23,864 INFO [Time-limited test] hbase.HBaseTestingUtility(3356): All regions for table multi_test_table assigned. 2018-06-21 23:23:23,865 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [30,000] milli-secs(wait.for.ratio=[1]) 2018-06-21 23:23:23,876 INFO [Time-limited test] zookeeper.ReadOnlyZKClient(139): Connect 0x4cb09ea8 to localhost:58989 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-06-21 23:23:23,891 DEBUG [Time-limited test] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34b76f1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-06-21 23:23:24,003 INFO [Time-limited test] hbase.ResourceChecker(148): before: client.TestMultiParallel#testBatchWithIncrementAndAppend Thread=580, OpenFileDescriptor=2434, MaxFileDescriptor=60000, SystemLoadAverage=523, ProcessCount=269, AvailableMemoryMB=13616 2018-06-21 23:23:24,003 WARN [Time-limited test] hbase.ResourceChecker(135): Thread=580 is superior to 500 2018-06-21 23:23:24,004 WARN [Time-limited test] hbase.ResourceChecker(135): OpenFileDescriptor=2434 is superior to 1024 2018-06-21 23:23:24,004 INFO [Time-limited test] client.TestMultiParallel(117): before 2018-06-21 23:23:24,004 INFO [Time-limited test] client.TestMultiParallel(133): before done 2018-06-21 23:23:24,004 INFO [Time-limited test] client.TestMultiParallel(492): test=testBatchWithIncrementAndAppend 2018-06-21 23:23:24,022 INFO [RS-EventLoopGroup-1-8] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-06-21 23:23:24,023 INFO [RS-EventLoopGroup-1-8] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 40712 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:24,111 INFO [Time-limited test] hbase.ResourceChecker(172): after: client.TestMultiParallel#testBatchWithIncrementAndAppend Thread=584 (was 580) Potentially hanging thread: Default-IPC-NioEventLoopGroup-9-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-shared-pool15-t3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-1-8 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-shared-pool15-t4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) - Thread LEAK? -, OpenFileDescriptor=2436 (was 2434) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=523 (was 523), ProcessCount=269 (was 269), AvailableMemoryMB=13611 (was 13616) 2018-06-21 23:23:24,111 WARN [Time-limited test] hbase.ResourceChecker(135): Thread=584 is superior to 500 2018-06-21 23:23:24,111 WARN [Time-limited test] hbase.ResourceChecker(135): OpenFileDescriptor=2436 is superior to 1024 2018-06-21 23:23:24,151 INFO [Time-limited test] hbase.ResourceChecker(148): before: client.TestMultiParallel#testBatchWithGet Thread=584, OpenFileDescriptor=2436, MaxFileDescriptor=60000, SystemLoadAverage=523, ProcessCount=269, AvailableMemoryMB=13611 2018-06-21 23:23:24,151 WARN [Time-limited test] hbase.ResourceChecker(135): Thread=584 is superior to 500 2018-06-21 23:23:24,152 WARN [Time-limited test] hbase.ResourceChecker(135): OpenFileDescriptor=2436 is superior to 1024 2018-06-21 23:23:24,152 INFO [Time-limited test] client.TestMultiParallel(117): before 2018-06-21 23:23:24,152 INFO [Time-limited test] client.TestMultiParallel(133): before done 2018-06-21 23:23:24,152 INFO [Time-limited test] client.TestMultiParallel(207): test=testBatchWithGet 2018-06-21 23:23:24,323 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-06-21 23:23:24,323 INFO [RS-EventLoopGroup-6-3] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-06-21 23:23:24,325 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 51559 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:24,328 INFO [RS-EventLoopGroup-6-3] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 56491 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:24,332 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-06-21 23:23:24,333 INFO [RS-EventLoopGroup-7-16] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-06-21 23:23:24,394 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 59350 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:24,395 INFO [RS-EventLoopGroup-7-16] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 43024 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:24,556 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-06-21 23:23:24,559 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-06-21 23:23:24,559 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-06-21 23:23:24,559 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-06-21 23:23:24,559 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-06-21 23:23:24,688 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-06-21 23:23:24,967 INFO [Time-limited test] hbase.ResourceChecker(172): after: client.TestMultiParallel#testBatchWithGet Thread=628 (was 584) Potentially hanging thread: RS-EventLoopGroup-5-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t24 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-shared-pool15-t16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-shared-pool15-t10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-shared-pool15-t7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-9-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-shared-pool15-t5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-9-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-9-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-shared-pool15-t13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-shared-pool15-t8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-6-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-3-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t22 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-shared-pool15-t15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-shared-pool15-t14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t23 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t20 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t26 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-shared-pool15-t12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-shared-pool15-t6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-shared-pool15-t9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-9-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t19 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-shared-pool15-t11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t21 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) - Thread LEAK? -, OpenFileDescriptor=2444 (was 2436) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=523 (was 523), ProcessCount=269 (was 269), AvailableMemoryMB=13510 (was 13611) 2018-06-21 23:23:24,968 WARN [Time-limited test] hbase.ResourceChecker(135): Thread=628 is superior to 500 2018-06-21 23:23:24,968 WARN [Time-limited test] hbase.ResourceChecker(135): OpenFileDescriptor=2444 is superior to 1024 2018-06-21 23:23:25,013 INFO [Time-limited test] hbase.ResourceChecker(148): before: client.TestMultiParallel#testBatchWithPut Thread=628, OpenFileDescriptor=2444, MaxFileDescriptor=60000, SystemLoadAverage=523, ProcessCount=269, AvailableMemoryMB=13509 2018-06-21 23:23:25,013 WARN [Time-limited test] hbase.ResourceChecker(135): Thread=628 is superior to 500 2018-06-21 23:23:25,013 WARN [Time-limited test] hbase.ResourceChecker(135): OpenFileDescriptor=2444 is superior to 1024 2018-06-21 23:23:25,014 INFO [Time-limited test] client.TestMultiParallel(117): before 2018-06-21 23:23:25,014 INFO [Time-limited test] client.TestMultiParallel(133): before done 2018-06-21 23:23:25,014 INFO [Time-limited test] client.TestMultiParallel(359): test=testBatchWithPut 2018-06-21 23:23:25,028 INFO [RS-EventLoopGroup-4-5] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-06-21 23:23:25,028 INFO [RS-EventLoopGroup-4-5] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 45776 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:25,135 INFO [RS-EventLoopGroup-7-27] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-06-21 23:23:25,138 INFO [RS-EventLoopGroup-7-27] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 43058 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:25,138 INFO [RS-EventLoopGroup-5-4] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-06-21 23:23:25,138 INFO [RS-EventLoopGroup-5-4] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 59387 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:25,139 INFO [RS-EventLoopGroup-3-4] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-06-21 23:23:25,138 INFO [RS-EventLoopGroup-6-4] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-06-21 23:23:25,139 INFO [RS-EventLoopGroup-3-4] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 51598 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:25,139 INFO [RS-EventLoopGroup-1-9] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-06-21 23:23:25,140 INFO [RS-EventLoopGroup-1-9] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 40751 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:25,140 INFO [RS-EventLoopGroup-6-4] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 56528 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:25,168 ERROR [Time-limited test] helpers.MarkerIgnoringBase(159): ***** ABORTING region server asf911.gq1.ygridcore.net,59970,1529623390066: Aborting for tests ***** java.lang.Exception: testBatchWithPut at org.apache.hadoop.hbase.client.TestMultiParallel.testBatchWithPut(TestMultiParallel.java:373) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) 2018-06-21 23:23:25,169 ERROR [Time-limited test] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.client.TestMultiParallel$MyMasterObserver, org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-06-21 23:23:25,197 INFO [Time-limited test] regionserver.HRegionServer(2371): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 1312292864, "init" : 792723456, "max" : 2609905664, "used" : 327792688 }, "NonHeapMemoryUsage" : { "committed" : 124633088, "init" : 2555904, "max" : -1, "used" : 122639024 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numActiveGeneralHandler" : 0, "numActivePriorityHandler" : 0, "numActiveReplicationHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 5220, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 30, "TotalCallTime_min" : 0, "TotalCallTime_max" : 37, "TotalCallTime_mean" : 6, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 3, "TotalCallTime_75th_percentile" : 6, "TotalCallTime_90th_percentile" : 12, "TotalCallTime_95th_percentile" : 35, "TotalCallTime_98th_percentile" : 37, "TotalCallTime_99th_percentile" : 37, "TotalCallTime_99.9th_percentile" : 37, "TotalCallTime_TimeRangeCount_0-1" : 13, "TotalCallTime_TimeRangeCount_1-3" : 4, "TotalCallTime_TimeRangeCount_3-10" : 9, "TotalCallTime_TimeRangeCount_10-30" : 2, "TotalCallTime_TimeRangeCount_30-100" : 2, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 30, "ResponseSize_min" : 0, "ResponseSize_max" : 823, "ResponseSize_mean" : 75, "ResponseSize_25th_percentile" : 2, "ResponseSize_median" : 4, "ResponseSize_75th_percentile" : 57, "ResponseSize_90th_percentile" : 58, "ResponseSize_95th_percentile" : 397, "ResponseSize_98th_percentile" : 652, "ResponseSize_99th_percentile" : 737, "ResponseSize_99.9th_percentile" : 814, "ResponseSize_SizeRangeCount_0-10" : 16, "ResponseSize_SizeRangeCount_10-100" : 11, "ResponseSize_SizeRangeCount_100-1000" : 3, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 0, "ProcessCallTime_num_ops" : 30, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 37, "ProcessCallTime_mean" : 5, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 3, "ProcessCallTime_75th_percentile" : 6, "ProcessCallTime_90th_percentile" : 12, "ProcessCallTime_95th_percentile" : 35, "ProcessCallTime_98th_percentile" : 37, "ProcessCallTime_99th_percentile" : 37, "ProcessCallTime_99.9th_percentile" : 37, "ProcessCallTime_TimeRangeCount_0-1" : 13, "ProcessCallTime_TimeRangeCount_1-3" : 4, "ProcessCallTime_TimeRangeCount_3-10" : 9, "ProcessCallTime_TimeRangeCount_10-30" : 2, "ProcessCallTime_TimeRangeCount_30-100" : 2, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 5, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 30, "RequestSize_min" : 78, "RequestSize_max" : 338, "RequestSize_mean" : 120, "RequestSize_25th_percentile" : 90, "RequestSize_median" : 93, "RequestSize_75th_percentile" : 117, "RequestSize_90th_percentile" : 152, "RequestSize_95th_percentile" : 276, "RequestSize_98th_percentile" : 313, "RequestSize_99th_percentile" : 325, "RequestSize_99.9th_percentile" : 336, "RequestSize_SizeRangeCount_10-100" : 19, "RequestSize_SizeRangeCount_100-1000" : 11, "sentBytes" : 2560, "QueueCallTime_num_ops" : 30, "QueueCallTime_min" : 0, "QueueCallTime_max" : 1, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 1, "QueueCallTime_95th_percentile" : 1, "QueueCallTime_98th_percentile" : 1, "QueueCallTime_99th_percentile" : 1, "QueueCallTime_99.9th_percentile" : 1, "QueueCallTime_TimeRangeCount_0-1" : 30, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:58989", "tag.serverName" : "asf911.gq1.ygridcore.net,58235,1529623390382", "tag.clusterId" : "2ed7179d-84c9-432a-b6b5-556343aa7690", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "regionCount" : 1, "storeCount" : 1, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 0, "memStoreSize" : 0, "storeFileSize" : 0, "maxStoreFileAge" : 0, "minStoreFileAge" : 0, "avgStoreFileAge" : 0, "numReferenceFiles" : 0, "regionServerStartTime" : 1529623390382, "averageRegionSize" : 0, "storeFileIndexSize" : 0, "staticIndexSize" : 0, "staticBloomSize" : 0, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1043196664, "blockCacheCount" : 0, "blockCacheSize" : 765640, "blockCacheCountHitPercent" : 0.0, "blockCacheExpressHitPercent" : 0.0, "l1CacheHitCount" : 0, "l1CacheMissCount" : 0, "l1CacheHitRatio" : 0.0, "l1CacheMissRatio" : 0.0, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "mobFileCacheCount" : 0, "mobFileCacheHitPercent" : 0.0, "totalRequestCount" : 8, "totalRowActionRequestCount" : 0, "readRequestCount" : 0, "filteredReadRequestCount" : 0, "writeRequestCount" : 0, "rpcGetRequestCount" : 4, "rpcScanRequestCount" : 1, "rpcMultiRequestCount" : 0, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "blockCacheHitCount" : 0, "blockCacheHitCountPrimary" : 0, "blockCacheMissCount" : 0, "blockCacheMissCountPrimary" : 0, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 0, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 0, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "updatesBlockedTime" : 0, "flushedCellsCount" : 0, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 0, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 1, "ScanTime_min" : 1, "ScanTime_max" : 1, "ScanTime_mean" : 1, "ScanTime_25th_percentile" : 1, "ScanTime_median" : 1, "ScanTime_75th_percentile" : 1, "ScanTime_90th_percentile" : 1, "ScanTime_95th_percentile" : 1, "ScanTime_98th_percentile" : 1, "ScanTime_99th_percentile" : 1, "ScanTime_99.9th_percentile" : 1, "ScanTime_TimeRangeCount_0-1" : 1, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 3, "Put_max" : 5, "Put_mean" : 4, "Put_25th_percentile" : 3, "Put_median" : 4, "Put_75th_percentile" : 5, "Put_90th_percentile" : 5, "Put_95th_percentile" : 5, "Put_98th_percentile" : 5, "Put_99th_percentile" : 5, "Put_99.9th_percentile" : 5, "Put_TimeRangeCount_1-3" : 1, "Put_TimeRangeCount_3-10" : 1, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 0, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 0, "PutBatch_min" : 0, "PutBatch_max" : 0, "PutBatch_mean" : 0, "PutBatch_25th_percentile" : 0, "PutBatch_median" : 0, "PutBatch_75th_percentile" : 0, "PutBatch_90th_percentile" : 0, "PutBatch_95th_percentile" : 0, "PutBatch_98th_percentile" : 0, "PutBatch_99th_percentile" : 0, "PutBatch_99.9th_percentile" : 0, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 4, "Get_min" : 0, "Get_max" : 0, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 0, "Get_95th_percentile" : 0, "Get_98th_percentile" : 0, "Get_99th_percentile" : 0, "Get_99.9th_percentile" : 0, "Get_TimeRangeCount_0-1" : 4, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 0, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 0, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 1, "ScanSize_min" : 86, "ScanSize_max" : 86, "ScanSize_mean" : 86, "ScanSize_25th_percentile" : 86, "ScanSize_median" : 86, "ScanSize_75th_percentile" : 86, "ScanSize_90th_percentile" : 86, "ScanSize_95th_percentile" : 86, "ScanSize_98th_percentile" : 86, "ScanSize_99th_percentile" : 86, "ScanSize_99.9th_percentile" : 86, "ScanSize_SizeRangeCount_10-100" : 1, "slowGetCount" : 0, "flushedMemstoreBytes" : 0, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 0, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-06-21 23:23:25,205 ERROR [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=55146] master.MasterRpcServices(509): Region server asf911.gq1.ygridcore.net,59970,1529623390066 reported a fatal error: ***** ABORTING region server asf911.gq1.ygridcore.net,59970,1529623390066: Aborting for tests ***** Cause: java.lang.Exception: testBatchWithPut at org.apache.hadoop.hbase.client.TestMultiParallel.testBatchWithPut(TestMultiParallel.java:373) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) 2018-06-21 23:23:25,209 INFO [Time-limited test] regionserver.HRegionServer(2148): ***** STOPPING region server 'asf911.gq1.ygridcore.net,59970,1529623390066' ***** 2018-06-21 23:23:25,209 INFO [Time-limited test] regionserver.HRegionServer(2162): STOPPED: Aborting for tests 2018-06-21 23:23:25,209 INFO [RS:0;asf911:59970] regionserver.SplitLogWorker(166): Sending interrupt to stop the worker thread 2018-06-21 23:23:25,210 INFO [SplitLogWorker-asf911:59970] regionserver.SplitLogWorker(148): SplitLogWorker interrupted. Exiting. 2018-06-21 23:23:25,210 INFO [RS:0;asf911:59970] regionserver.HeapMemoryManager(221): Stopping 2018-06-21 23:23:25,210 INFO [SplitLogWorker-asf911:59970] regionserver.SplitLogWorker(157): SplitLogWorker asf911.gq1.ygridcore.net,59970,1529623390066 exiting 2018-06-21 23:23:25,210 INFO [RS:0;asf911:59970] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager abruptly. 2018-06-21 23:23:25,213 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(383): MemStoreFlusher.0 exiting 2018-06-21 23:23:25,210 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(383): MemStoreFlusher.1 exiting 2018-06-21 23:23:25,213 INFO [RS:0;asf911:59970] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager abruptly. 2018-06-21 23:23:25,232 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1520): Closing 28b5f974b61112896f4c4f1127dc2b99, disabling compactions & flushes 2018-06-21 23:23:25,232 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1560): Updates disabled for region multi_test_table,mmm,1529623399386.28b5f974b61112896f4c4f1127dc2b99. 2018-06-21 23:23:25,232 INFO [RS:0;asf911:59970] regionserver.HRegionServer(1066): aborting server asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:25,233 DEBUG [RS:0;asf911:59970] zookeeper.MetaTableLocator(642): Stopping MetaTableLocator 2018-06-21 23:23:25,234 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1520): Closing 8e2537011449fca6295a357cbde76e6c, disabling compactions & flushes 2018-06-21 23:23:25,234 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1520): Closing 573e96916daa12dfc083e39a253dc3a2, disabling compactions & flushes 2018-06-21 23:23:25,235 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1560): Updates disabled for region multi_test_table,jjj,1529623399386.573e96916daa12dfc083e39a253dc3a2. 2018-06-21 23:23:25,235 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1560): Updates disabled for region multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c. 2018-06-21 23:23:25,235 INFO [RS:0;asf911:59970] zookeeper.ReadOnlyZKClient(350): Close zookeeper connection 0x125d969b to localhost:58989 2018-06-21 23:23:25,235 ERROR [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1661): Memstore data size is 1092 2018-06-21 23:23:25,235 DEBUG [RS:0;asf911:59970] ipc.AbstractRpcClient(483): Stopping rpc client 2018-06-21 23:23:25,236 INFO [RS:0;asf911:59970] regionserver.HRegionServer(1399): Waiting on 5 regions to close 2018-06-21 23:23:25,236 ERROR [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1661): Memstore data size is 1092 2018-06-21 23:23:25,236 ERROR [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1661): Memstore data size is 1092 2018-06-21 23:23:25,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=59970] ipc.CallRunner(142): callId: 55 service: ClientService methodName: Multi size: 4.7 K connection: 67.195.81.155:51598 deadline: 1529623465233, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server asf911.gq1.ygridcore.net,59970,1529623390066 aborting 2018-06-21 23:23:25,236 DEBUG [RS:0;asf911:59970] regionserver.HRegionServer(1403): Online Regions={28b5f974b61112896f4c4f1127dc2b99=multi_test_table,mmm,1529623399386.28b5f974b61112896f4c4f1127dc2b99., 8e2537011449fca6295a357cbde76e6c=multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c., 573e96916daa12dfc083e39a253dc3a2=multi_test_table,jjj,1529623399386.573e96916daa12dfc083e39a253dc3a2., 0dfb3ebb8c759d632ec793ac0387ec95=multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95., 794028a2b8fc0a516890763306629d4c=multi_test_table,zzz,1529623399386.794028a2b8fc0a516890763306629d4c.} 2018-06-21 23:23:25,238 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1674): Closed multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c. 2018-06-21 23:23:25,238 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1674): Closed multi_test_table,jjj,1529623399386.573e96916daa12dfc083e39a253dc3a2. 2018-06-21 23:23:25,238 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] handler.CloseRegionHandler(124): Closed multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c. 2018-06-21 23:23:25,238 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1674): Closed multi_test_table,mmm,1529623399386.28b5f974b61112896f4c4f1127dc2b99. 2018-06-21 23:23:25,241 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1520): Closing 0dfb3ebb8c759d632ec793ac0387ec95, disabling compactions & flushes 2018-06-21 23:23:25,238 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] handler.CloseRegionHandler(124): Closed multi_test_table,jjj,1529623399386.573e96916daa12dfc083e39a253dc3a2. 2018-06-21 23:23:25,241 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1560): Updates disabled for region multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95. 2018-06-21 23:23:25,241 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] handler.CloseRegionHandler(124): Closed multi_test_table,mmm,1529623399386.28b5f974b61112896f4c4f1127dc2b99. 2018-06-21 23:23:25,242 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1520): Closing 794028a2b8fc0a516890763306629d4c, disabling compactions & flushes 2018-06-21 23:23:25,245 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1560): Updates disabled for region multi_test_table,zzz,1529623399386.794028a2b8fc0a516890763306629d4c. 2018-06-21 23:23:25,246 ERROR [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1661): Memstore data size is 432 2018-06-21 23:23:25,246 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1674): Closed multi_test_table,zzz,1529623399386.794028a2b8fc0a516890763306629d4c. 2018-06-21 23:23:25,246 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] handler.CloseRegionHandler(124): Closed multi_test_table,zzz,1529623399386.794028a2b8fc0a516890763306629d4c. 2018-06-21 23:23:25,247 ERROR [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1661): Memstore data size is 1092 2018-06-21 23:23:25,247 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1674): Closed multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95. 2018-06-21 23:23:25,248 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] handler.CloseRegionHandler(124): Closed multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95. 2018-06-21 23:23:25,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=59970] ipc.CallRunner(142): callId: 69 service: ClientService methodName: Multi size: 4.7 K connection: 67.195.81.155:51598 deadline: 1529623465404, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server asf911.gq1.ygridcore.net,59970,1529623390066 aborting 2018-06-21 23:23:25,437 INFO [RS:0;asf911:59970] regionserver.HRegionServer(1097): stopping server asf911.gq1.ygridcore.net,59970,1529623390066; all regions closed. 2018-06-21 23:23:25,447 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741833_1009{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW]]} size 0 2018-06-21 23:23:25,448 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741833_1009{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW]]} size 0 2018-06-21 23:23:25,448 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741833_1009{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW]]} size 0 2018-06-21 23:23:25,451 DEBUG [RS:0;asf911:59970] ipc.AbstractRpcClient(483): Stopping rpc client 2018-06-21 23:23:25,452 INFO [RS:0;asf911:59970] regionserver.Leases(149): Closed leases 2018-06-21 23:23:25,453 INFO [RS:0;asf911:59970] hbase.ChoreService(327): Chore service for: regionserver/asf911:0 had [[ScheduledChore: Name: MovedRegionsCleaner for region asf911.gq1.ygridcore.net,59970,1529623390066 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS]] on shutdown 2018-06-21 23:23:25,453 INFO [regionserver/asf911:0.logRoller] regionserver.LogRoller(212): LogRoller exiting. 2018-06-21 23:23:25,453 INFO [RS:0;asf911:59970] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-06-21 23:23:25,453 INFO [RS:0;asf911:59970] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-06-21 23:23:25,453 INFO [RS:0;asf911:59970] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-06-21 23:23:25,456 INFO [RS:0;asf911:59970] ipc.NettyRpcServer(143): Stopping server on /67.195.81.155:59970 2018-06-21 23:23:25,475 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:59970-0x16424a89dd40001, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:25,475 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:25,475 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:59970-0x16424a89dd40001, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:25,475 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:25,475 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:25,475 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:25,475 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,59970,1529623390066 2018-06-21 23:23:25,483 INFO [RS:0;asf911:59970] regionserver.HRegionServer(1153): Exiting; stopping=asf911.gq1.ygridcore.net,59970,1529623390066; zookeeper connection closed. 2018-06-21 23:23:25,484 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@250a1cb] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(221): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@250a1cb 2018-06-21 23:23:25,484 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:25,484 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:25,484 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:25,484 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:25,485 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:25,485 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:25,485 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:25,484 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:25,485 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:25,485 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:25,485 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:25,486 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:25,486 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:25,486 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:25,486 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:25,486 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:25,487 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:25,487 INFO [Time-limited test-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(122): /hbase/rs/asf911.gq1.ygridcore.net,59970,1529623390066 znode expired, triggering replicatorRemoved event 2018-06-21 23:23:25,490 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:25,490 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:25,492 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:25,492 INFO [Time-limited test-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(122): /hbase/rs/asf911.gq1.ygridcore.net,59970,1529623390066 znode expired, triggering replicatorRemoved event 2018-06-21 23:23:25,492 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:25,493 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:25,493 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:25,493 INFO [Time-limited test-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(122): /hbase/rs/asf911.gq1.ygridcore.net,59970,1529623390066 znode expired, triggering replicatorRemoved event 2018-06-21 23:23:25,493 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:25,493 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:25,494 INFO [Time-limited test-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(122): /hbase/rs/asf911.gq1.ygridcore.net,59970,1529623390066 znode expired, triggering replicatorRemoved event 2018-06-21 23:23:25,494 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:25,495 INFO [RegionServerTracker-0] master.RegionServerTracker(159): RegionServer ephemeral node deleted, processing expiration [asf911.gq1.ygridcore.net,59970,1529623390066] 2018-06-21 23:23:25,495 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:25,496 INFO [RegionServerTracker-0] master.ServerManager(604): Processing expiration of asf911.gq1.ygridcore.net,59970,1529623390066 on asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:25,497 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:25,496 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:25,497 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:25,497 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:25,497 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:25,497 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:25,498 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:25,497 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:25,498 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:25,498 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:25,498 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:25,498 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:25,499 INFO [Time-limited test-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(122): /hbase/rs/asf911.gq1.ygridcore.net,59970,1529623390066 znode expired, triggering replicatorRemoved event 2018-06-21 23:23:25,499 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:25,498 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:25,499 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:25,499 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:25,499 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:25,499 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:25,499 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:25,500 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:25,500 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:25,500 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:25,500 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:25,500 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:25,501 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:25,501 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:25,502 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:25,502 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:25,608 INFO [regionserver/asf911:0.leaseChecker] regionserver.Leases(149): Closed leases 2018-06-21 23:23:25,631 DEBUG [Default-IPC-NioEventLoopGroup-9-14] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:59970 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:59970 2018-06-21 23:23:25,671 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(887): Stored pid=34, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure server=asf911.gq1.ygridcore.net,59970,1529623390066, splitWal=true, meta=false 2018-06-21 23:23:25,671 DEBUG [RegionServerTracker-0] assignment.AssignmentManager(1334): Added=asf911.gq1.ygridcore.net,59970,1529623390066 to dead servers, submitted shutdown handler to be executed meta=false 2018-06-21 23:23:25,676 INFO [PEWorker-8] procedure.ServerCrashProcedure(118): Start pid=34, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure server=asf911.gq1.ygridcore.net,59970,1529623390066, splitWal=true, meta=false 2018-06-21 23:23:25,808 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(239): Splitting WALs pid=34, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS; ServerCrashProcedure server=asf911.gq1.ygridcore.net,59970,1529623390066, splitWal=true, meta=false 2018-06-21 23:23:25,811 DEBUG [PEWorker-2] master.MasterWalManager(283): Renamed region directory: hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,59970,1529623390066-splitting 2018-06-21 23:23:25,811 INFO [PEWorker-2] master.SplitLogManager(461): dead splitlog workers [asf911.gq1.ygridcore.net,59970,1529623390066] 2018-06-21 23:23:25,814 INFO [PEWorker-2] master.SplitLogManager(241): Started splitting 1 logs in [hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,59970,1529623390066-splitting] for [asf911.gq1.ygridcore.net,59970,1529623390066] 2018-06-21 23:23:25,833 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:25,833 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:25,834 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:25,833 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:25,833 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:25,835 DEBUG [SplitLogWorker-asf911:52507] coordination.ZkSplitLogWorkerCoordination(436): Current region server asf911.gq1.ygridcore.net,52507,1529623390228 is ready to take more tasks, will get task list and try grab tasks again. 2018-06-21 23:23:25,835 DEBUG [SplitLogWorker-asf911:55146] coordination.ZkSplitLogWorkerCoordination(436): Current region server asf911.gq1.ygridcore.net,55146,1529623388534 is ready to take more tasks, will get task list and try grab tasks again. 2018-06-21 23:23:25,835 DEBUG [SplitLogWorker-asf911:56613] coordination.ZkSplitLogWorkerCoordination(436): Current region server asf911.gq1.ygridcore.net,56613,1529623390297 is ready to take more tasks, will get task list and try grab tasks again. 2018-06-21 23:23:25,835 DEBUG [SplitLogWorker-asf911:40767] coordination.ZkSplitLogWorkerCoordination(436): Current region server asf911.gq1.ygridcore.net,40767,1529623390148 is ready to take more tasks, will get task list and try grab tasks again. 2018-06-21 23:23:25,835 DEBUG [SplitLogWorker-asf911:58235] coordination.ZkSplitLogWorkerCoordination(436): Current region server asf911.gq1.ygridcore.net,58235,1529623390382 is ready to take more tasks, will get task list and try grab tasks again. 2018-06-21 23:23:25,836 DEBUG [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination(332): Put up splitlog task at znode /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59970%2C1529623390066-splitting%2Fasf911.gq1.ygridcore.net%252C59970%252C1529623390066.1529623393801 2018-06-21 23:23:25,851 INFO [SplitLogWorker-asf911:56613] coordination.ZkSplitLogWorkerCoordination(258): worker asf911.gq1.ygridcore.net,56613,1529623390297 acquired task /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59970%2C1529623390066-splitting%2Fasf911.gq1.ygridcore.net%252C59970%252C1529623390066.1529623393801 2018-06-21 23:23:25,860 INFO [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination(454): Task /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59970%2C1529623390066-splitting%2Fasf911.gq1.ygridcore.net%252C59970%252C1529623390066.1529623393801 acquired by asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:25,883 INFO [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] wal.WALSplitter(250): Splitting WAL=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,59970,1529623390066-splitting/asf911.gq1.ygridcore.net%2C59970%2C1529623390066.1529623393801, length=7542 2018-06-21 23:23:25,892 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59970%2C1529623390066-splitting%2Fasf911.gq1.ygridcore.net%252C59970%252C1529623390066.1529623393801 2018-06-21 23:23:25,892 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59970%2C1529623390066-splitting%2Fasf911.gq1.ygridcore.net%252C59970%252C1529623390066.1529623393801 2018-06-21 23:23:25,892 INFO [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] util.FSHDFSUtils(195): Recover lease on dfs file hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,59970,1529623390066-splitting/asf911.gq1.ygridcore.net%2C59970%2C1529623390066.1529623393801 2018-06-21 23:23:25,899 INFO [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] util.FSHDFSUtils(284): Recovered lease, attempt=0 on file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,59970,1529623390066-splitting/asf911.gq1.ygridcore.net%2C59970%2C1529623390066.1529623393801 after 7ms 2018-06-21 23:23:25,930 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] wal.WALSplitter(283): DLS Last flushed sequenceid for 8e2537011449fca6295a357cbde76e6c: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "family" sequence_id: 3 } 2018-06-21 23:23:25,932 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] wal.WALSplitter(283): DLS Last flushed sequenceid for 28b5f974b61112896f4c4f1127dc2b99: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "family" sequence_id: 3 } 2018-06-21 23:23:25,933 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] wal.WALSplitter(283): DLS Last flushed sequenceid for 794028a2b8fc0a516890763306629d4c: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "family" sequence_id: 3 } 2018-06-21 23:23:25,935 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] wal.WALSplitter(283): DLS Last flushed sequenceid for 573e96916daa12dfc083e39a253dc3a2: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "family" sequence_id: 3 } 2018-06-21 23:23:25,936 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] wal.WALSplitter(283): DLS Last flushed sequenceid for 0dfb3ebb8c759d632ec793ac0387ec95: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "family" sequence_id: 3 } 2018-06-21 23:23:25,945 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] wal.WALSplitter(335): Finishing writing output logs and closing down 2018-06-21 23:23:25,945 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] wal.WALSplitter$OutputSink(1162): Waiting for split writer threads to finish 2018-06-21 23:23:25,954 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0-Writer-0] wal.WALSplitter$LogRecoveredEditsOutputSink(1506): Creating writer path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/28b5f974b61112896f4c4f1127dc2b99/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C59970%2C1529623390066.1529623393801.temp 2018-06-21 23:23:25,954 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0-Writer-2] wal.WALSplitter$LogRecoveredEditsOutputSink(1506): Creating writer path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/573e96916daa12dfc083e39a253dc3a2/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C59970%2C1529623390066.1529623393801.temp 2018-06-21 23:23:25,954 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0-Writer-1] wal.WALSplitter$LogRecoveredEditsOutputSink(1506): Creating writer path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C59970%2C1529623390066.1529623393801.temp 2018-06-21 23:23:25,967 DEBUG [hconnection-0x3389a216-shared-pool17-t10] ipc.AbstractRpcClient(349): Not trying to connect to asf911.gq1.ygridcore.net/67.195.81.155:59970 this server is in the failed servers list 2018-06-21 23:23:25,974 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0-Writer-0] wal.WALSplitter$LogRecoveredEditsOutputSink(1506): Creating writer path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/794028a2b8fc0a516890763306629d4c/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C59970%2C1529623390066.1529623393801.temp 2018-06-21 23:23:25,974 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0-Writer-2] wal.WALSplitter$LogRecoveredEditsOutputSink(1506): Creating writer path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C59970%2C1529623390066.1529623393801.temp 2018-06-21 23:23:25,975 INFO [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] wal.WALSplitter$OutputSink(1186): 3 split writers finished; closing. 2018-06-21 23:23:26,002 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741868_1044{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW]]} size 0 2018-06-21 23:23:26,003 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741868_1044{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|FINALIZED]]} size 0 2018-06-21 23:23:26,003 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741866_1042{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|FINALIZED]]} size 0 2018-06-21 23:23:26,004 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741867_1043{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|FINALIZED]]} size 0 2018-06-21 23:23:26,004 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741867_1043{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|FINALIZED]]} size 0 2018-06-21 23:23:26,004 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741866_1042{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|FINALIZED], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:26,004 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741867_1043{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|FINALIZED]]} size 0 2018-06-21 23:23:26,005 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741866_1042{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|FINALIZED], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:26,005 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741868_1044{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|FINALIZED], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|FINALIZED]]} size 0 2018-06-21 23:23:26,006 DEBUG [split-log-closeStream-1] wal.WALSplitter$LogRecoveredEditsOutputSink(1381): Closed wap hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/28b5f974b61112896f4c4f1127dc2b99/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C59970%2C1529623390066.1529623393801.temp (wrote 2 edits, skipped 0 edits in 25ms 2018-06-21 23:23:26,007 DEBUG [split-log-closeStream-3] wal.WALSplitter$LogRecoveredEditsOutputSink(1381): Closed wap hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/573e96916daa12dfc083e39a253dc3a2/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C59970%2C1529623390066.1529623393801.temp (wrote 2 edits, skipped 0 edits in 28ms 2018-06-21 23:23:26,007 DEBUG [split-log-closeStream-2] wal.WALSplitter$LogRecoveredEditsOutputSink(1381): Closed wap hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C59970%2C1529623390066.1529623393801.temp (wrote 2 edits, skipped 0 edits in 21ms 2018-06-21 23:23:26,010 INFO [split-log-closeStream-3] wal.WALSplitter$LogRecoveredEditsOutputSink(1407): Rename hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/573e96916daa12dfc083e39a253dc3a2/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C59970%2C1529623390066.1529623393801.temp to hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/573e96916daa12dfc083e39a253dc3a2/recovered.edits/0000000000000000005 2018-06-21 23:23:26,010 INFO [split-log-closeStream-1] wal.WALSplitter$LogRecoveredEditsOutputSink(1407): Rename hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/28b5f974b61112896f4c4f1127dc2b99/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C59970%2C1529623390066.1529623393801.temp to hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/28b5f974b61112896f4c4f1127dc2b99/recovered.edits/0000000000000000005 2018-06-21 23:23:26,014 INFO [split-log-closeStream-2] wal.WALSplitter$LogRecoveredEditsOutputSink(1407): Rename hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C59970%2C1529623390066.1529623393801.temp to hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/recovered.edits/0000000000000000005 2018-06-21 23:23:26,045 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741869_1045{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|FINALIZED]]} size 0 2018-06-21 23:23:26,045 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741870_1046{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|FINALIZED]]} size 0 2018-06-21 23:23:26,045 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741869_1045{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|FINALIZED], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|FINALIZED]]} size 0 2018-06-21 23:23:26,045 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741870_1046{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|FINALIZED], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|FINALIZED]]} size 0 2018-06-21 23:23:26,046 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741869_1045{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|FINALIZED], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|FINALIZED]]} size 0 2018-06-21 23:23:26,046 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741870_1046{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|FINALIZED], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|FINALIZED]]} size 0 2018-06-21 23:23:26,047 DEBUG [split-log-closeStream-1] wal.WALSplitter$LogRecoveredEditsOutputSink(1381): Closed wap hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/794028a2b8fc0a516890763306629d4c/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C59970%2C1529623390066.1529623393801.temp (wrote 2 edits, skipped 0 edits in 8ms 2018-06-21 23:23:26,047 DEBUG [split-log-closeStream-3] wal.WALSplitter$LogRecoveredEditsOutputSink(1381): Closed wap hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C59970%2C1529623390066.1529623393801.temp (wrote 2 edits, skipped 0 edits in 11ms 2018-06-21 23:23:26,050 INFO [split-log-closeStream-1] wal.WALSplitter$LogRecoveredEditsOutputSink(1407): Rename hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/794028a2b8fc0a516890763306629d4c/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C59970%2C1529623390066.1529623393801.temp to hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/794028a2b8fc0a516890763306629d4c/recovered.edits/0000000000000000005 2018-06-21 23:23:26,050 INFO [split-log-closeStream-3] wal.WALSplitter$LogRecoveredEditsOutputSink(1407): Rename hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C59970%2C1529623390066.1529623393801.temp to hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/recovered.edits/0000000000000000005 2018-06-21 23:23:26,051 INFO [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] wal.WALSplitter(356): Processed 10 edits across 5 regions; edits skipped=5; log file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,59970,1529623390066-splitting/asf911.gq1.ygridcore.net%2C59970%2C1529623390066.1529623393801, length=7542, corrupted=false, progress failed=false 2018-06-21 23:23:26,058 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59970%2C1529623390066-splitting%2Fasf911.gq1.ygridcore.net%252C59970%252C1529623390066.1529623393801 2018-06-21 23:23:26,058 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59970%2C1529623390066-splitting%2Fasf911.gq1.ygridcore.net%252C59970%252C1529623390066.1529623393801 2018-06-21 23:23:26,058 INFO [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] coordination.ZkSplitLogWorkerCoordination(574): successfully transitioned task /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59970%2C1529623390066-splitting%2Fasf911.gq1.ygridcore.net%252C59970%252C1529623390066.1529623393801 to final state DONE asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:26,059 INFO [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] handler.WALSplitterHandler(99): Worker asf911.gq1.ygridcore.net,56613,1529623390297 done with task org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$ZkSplitTaskDetails@32e04cc1 in 205ms. Status = DONE 2018-06-21 23:23:26,059 INFO [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination(372): Task /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59970%2C1529623390066-splitting%2Fasf911.gq1.ygridcore.net%252C59970%252C1529623390066.1529623393801 entered state=DONE asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:26,108 INFO [Time-limited test-EventThread] wal.WALSplitter(448): Archived processed log hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,59970,1529623390066-splitting/asf911.gq1.ygridcore.net%2C59970%2C1529623390066.1529623393801 to hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/oldWALs/asf911.gq1.ygridcore.net%2C59970%2C1529623390066.1529623393801 2018-06-21 23:23:26,109 INFO [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination(415): Done splitting /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59970%2C1529623390066-splitting%2Fasf911.gq1.ygridcore.net%252C59970%252C1529623390066.1529623393801 2018-06-21 23:23:26,112 INFO [PEWorker-2] master.SplitLogManager(293): finished splitting (more than or equal to) 7542 bytes in 1 log files in [hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,59970,1529623390066-splitting] in 298ms 2018-06-21 23:23:26,112 DEBUG [PEWorker-2] procedure.ServerCrashProcedure(247): Done splitting WALs pid=34, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS; ServerCrashProcedure server=asf911.gq1.ygridcore.net,59970,1529623390066, splitWal=true, meta=false 2018-06-21 23:23:26,123 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59970%2C1529623390066-splitting%2Fasf911.gq1.ygridcore.net%252C59970%252C1529623390066.1529623393801 2018-06-21 23:23:26,124 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59970%2C1529623390066-splitting%2Fasf911.gq1.ygridcore.net%252C59970%252C1529623390066.1529623393801 2018-06-21 23:23:26,124 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:26,124 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:26,124 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:26,124 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:26,124 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:26,124 DEBUG [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination$DeleteAsyncCallback(696): Deleted /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C59970%2C1529623390066-splitting%2Fasf911.gq1.ygridcore.net%252C59970%252C1529623390066.1529623393801 2018-06-21 23:23:26,187 INFO [PEWorker-6] procedure2.ProcedureExecutor(1516): Initialized subprocedures=[{pid=35, ppid=34, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=573e96916daa12dfc083e39a253dc3a2}, {pid=36, ppid=34, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=28b5f974b61112896f4c4f1127dc2b99}, {pid=37, ppid=34, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=8e2537011449fca6295a357cbde76e6c}, {pid=38, ppid=34, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=0dfb3ebb8c759d632ec793ac0387ec95}, {pid=39, ppid=34, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=794028a2b8fc0a516890763306629d4c}] 2018-06-21 23:23:26,296 INFO [PEWorker-14] procedure.MasterProcedureScheduler(696): pid=39, ppid=34, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=794028a2b8fc0a516890763306629d4c checking lock on 794028a2b8fc0a516890763306629d4c 2018-06-21 23:23:26,297 INFO [PEWorker-15] procedure.MasterProcedureScheduler(696): pid=38, ppid=34, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=0dfb3ebb8c759d632ec793ac0387ec95 checking lock on 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:26,297 INFO [PEWorker-3] procedure.MasterProcedureScheduler(696): pid=35, ppid=34, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=573e96916daa12dfc083e39a253dc3a2 checking lock on 573e96916daa12dfc083e39a253dc3a2 2018-06-21 23:23:26,297 INFO [PEWorker-13] procedure.MasterProcedureScheduler(696): pid=37, ppid=34, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=8e2537011449fca6295a357cbde76e6c checking lock on 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:26,298 INFO [PEWorker-12] procedure.MasterProcedureScheduler(696): pid=36, ppid=34, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=28b5f974b61112896f4c4f1127dc2b99 checking lock on 28b5f974b61112896f4c4f1127dc2b99 2018-06-21 23:23:26,300 INFO [PEWorker-13] assignment.AssignProcedure(218): Starting pid=37, ppid=34, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=8e2537011449fca6295a357cbde76e6c; rit=OFFLINE, location=asf911.gq1.ygridcore.net,59970,1529623390066; forceNewPlan=false, retain=true 2018-06-21 23:23:26,300 INFO [PEWorker-15] assignment.AssignProcedure(218): Starting pid=38, ppid=34, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=0dfb3ebb8c759d632ec793ac0387ec95; rit=OFFLINE, location=asf911.gq1.ygridcore.net,59970,1529623390066; forceNewPlan=false, retain=true 2018-06-21 23:23:26,300 INFO [PEWorker-3] assignment.AssignProcedure(218): Starting pid=35, ppid=34, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=573e96916daa12dfc083e39a253dc3a2; rit=OFFLINE, location=asf911.gq1.ygridcore.net,59970,1529623390066; forceNewPlan=false, retain=true 2018-06-21 23:23:26,300 INFO [PEWorker-12] assignment.AssignProcedure(218): Starting pid=36, ppid=34, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=28b5f974b61112896f4c4f1127dc2b99; rit=OFFLINE, location=asf911.gq1.ygridcore.net,59970,1529623390066; forceNewPlan=false, retain=true 2018-06-21 23:23:26,300 INFO [PEWorker-14] assignment.AssignProcedure(218): Starting pid=39, ppid=34, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=794028a2b8fc0a516890763306629d4c; rit=OFFLINE, location=asf911.gq1.ygridcore.net,59970,1529623390066; forceNewPlan=false, retain=true 2018-06-21 23:23:26,454 INFO [master/asf911:0] balancer.BaseLoadBalancer(1497): Reassigned 5 regions. 0 retained the pre-restart assignment. 5 regions were assigned to random hosts, since the old hosts for these regions are no longer present in the cluster. These hosts were: 2018-06-21 23:23:26,454 INFO [PEWorker-5] assignment.RegionStateStore(198): pid=39 updating hbase:meta row=794028a2b8fc0a516890763306629d4c, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:26,454 INFO [PEWorker-11] assignment.RegionStateStore(198): pid=35 updating hbase:meta row=573e96916daa12dfc083e39a253dc3a2, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:26,455 INFO [PEWorker-4] assignment.RegionStateStore(198): pid=36 updating hbase:meta row=28b5f974b61112896f4c4f1127dc2b99, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:26,455 INFO [PEWorker-16] assignment.RegionStateStore(198): pid=37 updating hbase:meta row=8e2537011449fca6295a357cbde76e6c, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:26,455 INFO [PEWorker-10] assignment.RegionStateStore(198): pid=38 updating hbase:meta row=0dfb3ebb8c759d632ec793ac0387ec95, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:26,460 INFO [PEWorker-16] assignment.RegionTransitionProcedure(241): Dispatch pid=37, ppid=34, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=8e2537011449fca6295a357cbde76e6c; rit=OPENING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:26,460 INFO [PEWorker-5] assignment.RegionTransitionProcedure(241): Dispatch pid=39, ppid=34, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=794028a2b8fc0a516890763306629d4c; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:26,460 INFO [PEWorker-11] assignment.RegionTransitionProcedure(241): Dispatch pid=35, ppid=34, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=573e96916daa12dfc083e39a253dc3a2; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:26,460 INFO [PEWorker-10] assignment.RegionTransitionProcedure(241): Dispatch pid=38, ppid=34, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=0dfb3ebb8c759d632ec793ac0387ec95; rit=OPENING, location=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:26,460 INFO [PEWorker-4] assignment.RegionTransitionProcedure(241): Dispatch pid=36, ppid=34, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=28b5f974b61112896f4c4f1127dc2b99; rit=OPENING, location=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:26,495 DEBUG [hconnection-0x3389a216-shared-pool17-t10] ipc.AbstractRpcClient(349): Not trying to connect to asf911.gq1.ygridcore.net/67.195.81.155:59970 this server is in the failed servers list 2018-06-21 23:23:26,610 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(119): Fallback to compat rpc execution for serverName=asf911.gq1.ygridcore.net,55146,1529623388534 version=0 2018-06-21 23:23:26,612 INFO [RSProcedureDispatcher-pool3-t14] regionserver.RSRpcServices(1980): Open multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c. 2018-06-21 23:23:26,612 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=40767] regionserver.RSRpcServices(1980): Open multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95. 2018-06-21 23:23:26,613 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58235] regionserver.RSRpcServices(1980): Open multi_test_table,zzz,1529623399386.794028a2b8fc0a516890763306629d4c. 2018-06-21 23:23:26,613 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=56613] regionserver.RSRpcServices(1980): Open multi_test_table,mmm,1529623399386.28b5f974b61112896f4c4f1127dc2b99. 2018-06-21 23:23:26,613 DEBUG [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegion(7094): Opening region: {ENCODED => 8e2537011449fca6295a357cbde76e6c, NAME => 'multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c.', STARTKEY => 'rrr', ENDKEY => 'sss'} 2018-06-21 23:23:26,614 DEBUG [RS_OPEN_REGION-master/asf911:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:26,614 DEBUG [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegion(827): Instantiated multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:26,614 DEBUG [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegion(7134): checking encryption for 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:26,614 DEBUG [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegion(7139): checking classloading for 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:26,619 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58235] regionserver.RSRpcServices(1980): Open multi_test_table,jjj,1529623399386.573e96916daa12dfc083e39a253dc3a2. 2018-06-21 23:23:26,619 DEBUG [StoreOpener-8e2537011449fca6295a357cbde76e6c-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/family 2018-06-21 23:23:26,620 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7094): Opening region: {ENCODED => 573e96916daa12dfc083e39a253dc3a2, NAME => 'multi_test_table,jjj,1529623399386.573e96916daa12dfc083e39a253dc3a2.', STARTKEY => 'jjj', ENDKEY => 'kkk'} 2018-06-21 23:23:26,619 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7094): Opening region: {ENCODED => 794028a2b8fc0a516890763306629d4c, NAME => 'multi_test_table,zzz,1529623399386.794028a2b8fc0a516890763306629d4c.', STARTKEY => 'zzz', ENDKEY => ''} 2018-06-21 23:23:26,619 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7094): Opening region: {ENCODED => 0dfb3ebb8c759d632ec793ac0387ec95, NAME => 'multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95.', STARTKEY => 'vvv', ENDKEY => 'www'} 2018-06-21 23:23:26,619 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7094): Opening region: {ENCODED => 28b5f974b61112896f4c4f1127dc2b99, NAME => 'multi_test_table,mmm,1529623399386.28b5f974b61112896f4c4f1127dc2b99.', STARTKEY => 'mmm', ENDKEY => 'nnn'} 2018-06-21 23:23:26,620 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table 794028a2b8fc0a516890763306629d4c 2018-06-21 23:23:26,620 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table 573e96916daa12dfc083e39a253dc3a2 2018-06-21 23:23:26,621 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(827): Instantiated multi_test_table,jjj,1529623399386.573e96916daa12dfc083e39a253dc3a2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:26,620 DEBUG [StoreOpener-8e2537011449fca6295a357cbde76e6c-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/family 2018-06-21 23:23:26,621 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7134): checking encryption for 573e96916daa12dfc083e39a253dc3a2 2018-06-21 23:23:26,621 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table 28b5f974b61112896f4c4f1127dc2b99 2018-06-21 23:23:26,620 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(827): Instantiated multi_test_table,zzz,1529623399386.794028a2b8fc0a516890763306629d4c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:26,621 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7134): checking encryption for 794028a2b8fc0a516890763306629d4c 2018-06-21 23:23:26,622 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7139): checking classloading for 794028a2b8fc0a516890763306629d4c 2018-06-21 23:23:26,622 INFO [StoreOpener-8e2537011449fca6295a357cbde76e6c-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:26,620 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:26,621 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(827): Instantiated multi_test_table,mmm,1529623399386.28b5f974b61112896f4c4f1127dc2b99.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:26,622 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7134): checking encryption for 28b5f974b61112896f4c4f1127dc2b99 2018-06-21 23:23:26,623 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7139): checking classloading for 28b5f974b61112896f4c4f1127dc2b99 2018-06-21 23:23:26,621 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7139): checking classloading for 573e96916daa12dfc083e39a253dc3a2 2018-06-21 23:23:26,623 INFO [StoreOpener-8e2537011449fca6295a357cbde76e6c-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:26,622 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(827): Instantiated multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:26,624 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7134): checking encryption for 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:26,624 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7139): checking classloading for 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:26,624 INFO [StoreOpener-8e2537011449fca6295a357cbde76e6c-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:26,625 DEBUG [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegion(923): replaying wal for 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:26,627 DEBUG [StoreOpener-794028a2b8fc0a516890763306629d4c-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/794028a2b8fc0a516890763306629d4c/family 2018-06-21 23:23:26,627 DEBUG [StoreOpener-794028a2b8fc0a516890763306629d4c-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/794028a2b8fc0a516890763306629d4c/family 2018-06-21 23:23:26,627 DEBUG [StoreOpener-28b5f974b61112896f4c4f1127dc2b99-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/28b5f974b61112896f4c4f1127dc2b99/family 2018-06-21 23:23:26,627 DEBUG [StoreOpener-28b5f974b61112896f4c4f1127dc2b99-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/28b5f974b61112896f4c4f1127dc2b99/family 2018-06-21 23:23:26,628 INFO [StoreOpener-794028a2b8fc0a516890763306629d4c-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:26,628 DEBUG [StoreOpener-573e96916daa12dfc083e39a253dc3a2-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/573e96916daa12dfc083e39a253dc3a2/family 2018-06-21 23:23:26,628 DEBUG [StoreOpener-573e96916daa12dfc083e39a253dc3a2-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/573e96916daa12dfc083e39a253dc3a2/family 2018-06-21 23:23:26,628 INFO [StoreOpener-28b5f974b61112896f4c4f1127dc2b99-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:26,628 DEBUG [StoreOpener-0dfb3ebb8c759d632ec793ac0387ec95-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/family 2018-06-21 23:23:26,628 DEBUG [StoreOpener-0dfb3ebb8c759d632ec793ac0387ec95-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/family 2018-06-21 23:23:26,628 INFO [StoreOpener-573e96916daa12dfc083e39a253dc3a2-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:26,628 INFO [StoreOpener-28b5f974b61112896f4c4f1127dc2b99-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:26,629 INFO [StoreOpener-0dfb3ebb8c759d632ec793ac0387ec95-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:26,629 INFO [StoreOpener-573e96916daa12dfc083e39a253dc3a2-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:26,629 INFO [StoreOpener-0dfb3ebb8c759d632ec793ac0387ec95-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:26,629 DEBUG [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegion(4475): Found 1 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:26,629 INFO [StoreOpener-28b5f974b61112896f4c4f1127dc2b99-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:26,630 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(923): replaying wal for 28b5f974b61112896f4c4f1127dc2b99 2018-06-21 23:23:26,630 INFO [StoreOpener-0dfb3ebb8c759d632ec793ac0387ec95-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:26,630 INFO [StoreOpener-573e96916daa12dfc083e39a253dc3a2-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:26,630 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(923): replaying wal for 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:26,630 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(923): replaying wal for 573e96916daa12dfc083e39a253dc3a2 2018-06-21 23:23:26,631 INFO [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegion(4572): Replaying edits from hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/recovered.edits/0000000000000000005 2018-06-21 23:23:26,631 INFO [StoreOpener-794028a2b8fc0a516890763306629d4c-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:26,632 INFO [StoreOpener-794028a2b8fc0a516890763306629d4c-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:26,633 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(923): replaying wal for 794028a2b8fc0a516890763306629d4c 2018-06-21 23:23:26,633 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(4475): Found 1 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/28b5f974b61112896f4c4f1127dc2b99 2018-06-21 23:23:26,634 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(4475): Found 1 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/573e96916daa12dfc083e39a253dc3a2 2018-06-21 23:23:26,635 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(4572): Replaying edits from hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/28b5f974b61112896f4c4f1127dc2b99/recovered.edits/0000000000000000005 2018-06-21 23:23:26,635 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(4475): Found 1 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:26,635 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(4572): Replaying edits from hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/573e96916daa12dfc083e39a253dc3a2/recovered.edits/0000000000000000005 2018-06-21 23:23:26,636 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(4572): Replaying edits from hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/recovered.edits/0000000000000000005 2018-06-21 23:23:26,637 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(4475): Found 1 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/794028a2b8fc0a516890763306629d4c 2018-06-21 23:23:26,639 INFO [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(4572): Replaying edits from hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/794028a2b8fc0a516890763306629d4c/recovered.edits/0000000000000000005 2018-06-21 23:23:26,642 DEBUG [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegion(4760): Applied 28, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/recovered.edits/0000000000000000005 2018-06-21 23:23:26,642 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(4760): Applied 28, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/28b5f974b61112896f4c4f1127dc2b99/recovered.edits/0000000000000000005 2018-06-21 23:23:26,643 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(4760): Applied 28, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/573e96916daa12dfc083e39a253dc3a2/recovered.edits/0000000000000000005 2018-06-21 23:23:26,643 INFO [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=1.07 KB heapSize=2.19 KB 2018-06-21 23:23:26,643 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=1.07 KB heapSize=2.19 KB 2018-06-21 23:23:26,643 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=1.07 KB heapSize=2.19 KB 2018-06-21 23:23:26,645 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(4760): Applied 28, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/recovered.edits/0000000000000000005 2018-06-21 23:23:26,645 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=1.07 KB heapSize=2.19 KB 2018-06-21 23:23:26,652 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(4760): Applied 12, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=5, path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/794028a2b8fc0a516890763306629d4c/recovered.edits/0000000000000000005 2018-06-21 23:23:26,652 INFO [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=432 B heapSize=1.31 KB 2018-06-21 23:23:26,749 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741873_1049{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:26,749 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741873_1049{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|FINALIZED]]} size 0 2018-06-21 23:23:26,749 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741873_1049{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|FINALIZED]]} size 0 2018-06-21 23:23:26,752 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741872_1048{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|FINALIZED]]} size 0 2018-06-21 23:23:26,755 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741874_1050{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW]]} size 0 2018-06-21 23:23:26,755 INFO [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=432 B at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/794028a2b8fc0a516890763306629d4c/.tmp/family/c1f9b7329e644a4a82849f7da1d87f9f 2018-06-21 23:23:26,755 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741875_1051{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW]]} size 0 2018-06-21 23:23:26,756 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741871_1047{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW]]} size 0 2018-06-21 23:23:26,757 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741875_1051{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW]]} size 0 2018-06-21 23:23:26,757 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741871_1047{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW]]} size 0 2018-06-21 23:23:26,757 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741872_1048{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|FINALIZED]]} size 0 2018-06-21 23:23:26,757 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741874_1050{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW]]} size 0 2018-06-21 23:23:26,758 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=1.07 KB at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/.tmp/family/a9b2590c1fd74168ad2e74d2f3c0070d 2018-06-21 23:23:26,758 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=1.07 KB at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/28b5f974b61112896f4c4f1127dc2b99/.tmp/family/72f321129ef44ddbba21998cf228bb39 2018-06-21 23:23:26,758 INFO [RS_OPEN_REGION-master/asf911:0-0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=1.07 KB at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/.tmp/family/7f5cd2b2372e4de08c448cfcd864eded 2018-06-21 23:23:26,759 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=1.07 KB at sequenceid=5 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/573e96916daa12dfc083e39a253dc3a2/.tmp/family/0977964d88e84dc99b1073693ef0e65e 2018-06-21 23:23:26,760 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741875_1051 size 5632 2018-06-21 23:23:26,761 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741871_1047 size 5632 2018-06-21 23:23:26,765 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741872_1048 size 5632 2018-06-21 23:23:26,766 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741874_1050 size 5632 2018-06-21 23:23:26,833 DEBUG [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/.tmp/family/7f5cd2b2372e4de08c448cfcd864eded as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/family/7f5cd2b2372e4de08c448cfcd864eded 2018-06-21 23:23:26,833 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/573e96916daa12dfc083e39a253dc3a2/.tmp/family/0977964d88e84dc99b1073693ef0e65e as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/573e96916daa12dfc083e39a253dc3a2/family/0977964d88e84dc99b1073693ef0e65e 2018-06-21 23:23:26,833 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/794028a2b8fc0a516890763306629d4c/.tmp/family/c1f9b7329e644a4a82849f7da1d87f9f as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/794028a2b8fc0a516890763306629d4c/family/c1f9b7329e644a4a82849f7da1d87f9f 2018-06-21 23:23:26,833 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/28b5f974b61112896f4c4f1127dc2b99/.tmp/family/72f321129ef44ddbba21998cf228bb39 as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/28b5f974b61112896f4c4f1127dc2b99/family/72f321129ef44ddbba21998cf228bb39 2018-06-21 23:23:26,838 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/.tmp/family/a9b2590c1fd74168ad2e74d2f3c0070d as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/family/a9b2590c1fd74168ad2e74d2f3c0070d 2018-06-21 23:23:26,848 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/573e96916daa12dfc083e39a253dc3a2/family/0977964d88e84dc99b1073693ef0e65e, entries=20, sequenceid=5, filesize=5.5 K 2018-06-21 23:23:26,848 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/28b5f974b61112896f4c4f1127dc2b99/family/72f321129ef44ddbba21998cf228bb39, entries=20, sequenceid=5, filesize=5.5 K 2018-06-21 23:23:26,850 INFO [RS_OPEN_REGION-master/asf911:0-0] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/family/7f5cd2b2372e4de08c448cfcd864eded, entries=20, sequenceid=5, filesize=5.5 K 2018-06-21 23:23:26,852 INFO [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/794028a2b8fc0a516890763306629d4c/family/c1f9b7329e644a4a82849f7da1d87f9f, entries=12, sequenceid=5, filesize=5.1 K 2018-06-21 23:23:26,854 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/family/a9b2590c1fd74168ad2e74d2f3c0070d, entries=20, sequenceid=5, filesize=5.5 K 2018-06-21 23:23:26,860 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(2769): Finished flush of dataSize ~1.07 KB/1092, heapSize ~2.40 KB/2456, currentSize=0 B/0 for 0dfb3ebb8c759d632ec793ac0387ec95 in 214ms, sequenceid=5, compaction requested=false; wal=null 2018-06-21 23:23:26,860 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(2769): Finished flush of dataSize ~1.07 KB/1092, heapSize ~2.40 KB/2456, currentSize=0 B/0 for 28b5f974b61112896f4c4f1127dc2b99 in 216ms, sequenceid=5, compaction requested=false; wal=null 2018-06-21 23:23:26,860 INFO [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(2769): Finished flush of dataSize ~432 B/432, heapSize ~1.52 KB/1560, currentSize=0 B/0 for 794028a2b8fc0a516890763306629d4c in 208ms, sequenceid=5, compaction requested=false; wal=null 2018-06-21 23:23:26,860 INFO [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegion(2769): Finished flush of dataSize ~1.07 KB/1092, heapSize ~2.40 KB/2456, currentSize=0 B/0 for 8e2537011449fca6295a357cbde76e6c in 217ms, sequenceid=5, compaction requested=false; wal=null 2018-06-21 23:23:26,860 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(2769): Finished flush of dataSize ~1.07 KB/1092, heapSize ~2.40 KB/2456, currentSize=0 B/0 for 573e96916daa12dfc083e39a253dc3a2 in 216ms, sequenceid=5, compaction requested=false; wal=null 2018-06-21 23:23:26,863 INFO [IPC Server handler 4 on 56139] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741868_1044 127.0.0.1:42074 127.0.0.1:57095 127.0.0.1:54793 2018-06-21 23:23:26,863 INFO [IPC Server handler 8 on 56139] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741866_1042 127.0.0.1:57095 127.0.0.1:33402 127.0.0.1:53553 2018-06-21 23:23:26,863 DEBUG [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegion(4552): Deleted recovered.edits file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/recovered.edits/0000000000000000005 2018-06-21 23:23:26,863 INFO [IPC Server handler 9 on 56139] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741867_1043 127.0.0.1:53553 127.0.0.1:54793 127.0.0.1:33402 2018-06-21 23:23:26,863 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(4552): Deleted recovered.edits file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/28b5f974b61112896f4c4f1127dc2b99/recovered.edits/0000000000000000005 2018-06-21 23:23:26,864 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(931): stopping wal replay for 28b5f974b61112896f4c4f1127dc2b99 2018-06-21 23:23:26,863 DEBUG [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegion(931): stopping wal replay for 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:26,864 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(4552): Deleted recovered.edits file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/573e96916daa12dfc083e39a253dc3a2/recovered.edits/0000000000000000005 2018-06-21 23:23:26,864 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(931): stopping wal replay for 573e96916daa12dfc083e39a253dc3a2 2018-06-21 23:23:26,864 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(943): Cleaning up temporary data for 573e96916daa12dfc083e39a253dc3a2 2018-06-21 23:23:26,864 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(943): Cleaning up temporary data for 28b5f974b61112896f4c4f1127dc2b99 2018-06-21 23:23:26,864 INFO [IPC Server handler 6 on 56139] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741869_1045 127.0.0.1:42074 127.0.0.1:33402 127.0.0.1:53553 2018-06-21 23:23:26,864 DEBUG [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegion(943): Cleaning up temporary data for 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:26,865 INFO [IPC Server handler 1 on 56139] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741870_1046 127.0.0.1:33402 127.0.0.1:54793 127.0.0.1:42074 2018-06-21 23:23:26,865 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(4552): Deleted recovered.edits file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/recovered.edits/0000000000000000005 2018-06-21 23:23:26,865 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(931): stopping wal replay for 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:26,865 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(943): Cleaning up temporary data for 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:26,865 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(4552): Deleted recovered.edits file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/794028a2b8fc0a516890763306629d4c/recovered.edits/0000000000000000005 2018-06-21 23:23:26,865 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(931): stopping wal replay for 794028a2b8fc0a516890763306629d4c 2018-06-21 23:23:26,865 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(943): Cleaning up temporary data for 794028a2b8fc0a516890763306629d4c 2018-06-21 23:23:26,865 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(954): Cleaning up detritus for 573e96916daa12dfc083e39a253dc3a2 2018-06-21 23:23:26,866 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(954): Cleaning up detritus for 28b5f974b61112896f4c4f1127dc2b99 2018-06-21 23:23:26,866 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(954): Cleaning up detritus for 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:26,866 DEBUG [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegion(954): Cleaning up detritus for 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:26,866 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(954): Cleaning up detritus for 794028a2b8fc0a516890763306629d4c 2018-06-21 23:23:26,868 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(976): writing seq id for 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:26,868 DEBUG [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegion(976): writing seq id for 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:26,868 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(976): writing seq id for 573e96916daa12dfc083e39a253dc3a2 2018-06-21 23:23:26,869 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(976): writing seq id for 794028a2b8fc0a516890763306629d4c 2018-06-21 23:23:26,869 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(976): writing seq id for 28b5f974b61112896f4c4f1127dc2b99 2018-06-21 23:23:26,873 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2018-06-21 23:23:26,873 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/573e96916daa12dfc083e39a253dc3a2/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2018-06-21 23:23:26,874 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/794028a2b8fc0a516890763306629d4c/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2018-06-21 23:23:26,874 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/28b5f974b61112896f4c4f1127dc2b99/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2018-06-21 23:23:26,874 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(980): Opened 0dfb3ebb8c759d632ec793ac0387ec95; next sequenceid=6 2018-06-21 23:23:26,874 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(987): Running coprocessor post-open hooks for 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:26,874 DEBUG [RS_OPEN_REGION-master/asf911:0-0] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/recovered.edits/5.seqid, newMaxSeqId=5, maxSeqId=1 2018-06-21 23:23:26,874 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(980): Opened 573e96916daa12dfc083e39a253dc3a2; next sequenceid=6 2018-06-21 23:23:26,874 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(987): Running coprocessor post-open hooks for 573e96916daa12dfc083e39a253dc3a2 2018-06-21 23:23:26,875 INFO [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(980): Opened 794028a2b8fc0a516890763306629d4c; next sequenceid=6 2018-06-21 23:23:26,875 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(980): Opened 28b5f974b61112896f4c4f1127dc2b99; next sequenceid=6 2018-06-21 23:23:26,875 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(987): Running coprocessor post-open hooks for 28b5f974b61112896f4c4f1127dc2b99 2018-06-21 23:23:26,875 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(987): Running coprocessor post-open hooks for 794028a2b8fc0a516890763306629d4c 2018-06-21 23:23:26,875 INFO [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegion(980): Opened 8e2537011449fca6295a357cbde76e6c; next sequenceid=6 2018-06-21 23:23:26,875 DEBUG [RS_OPEN_REGION-master/asf911:0-0] regionserver.HRegion(987): Running coprocessor post-open hooks for 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:26,876 INFO [PostOpenDeployTasks:0dfb3ebb8c759d632ec793ac0387ec95] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95. 2018-06-21 23:23:26,877 INFO [PostOpenDeployTasks:28b5f974b61112896f4c4f1127dc2b99] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,mmm,1529623399386.28b5f974b61112896f4c4f1127dc2b99. 2018-06-21 23:23:26,887 INFO [PostOpenDeployTasks:573e96916daa12dfc083e39a253dc3a2] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,jjj,1529623399386.573e96916daa12dfc083e39a253dc3a2. 2018-06-21 23:23:26,887 INFO [PostOpenDeployTasks:8e2537011449fca6295a357cbde76e6c] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c. 2018-06-21 23:23:26,889 INFO [PostOpenDeployTasks:794028a2b8fc0a516890763306629d4c] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,zzz,1529623399386.794028a2b8fc0a516890763306629d4c. 2018-06-21 23:23:26,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=6, pid=36, ppid=34, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=28b5f974b61112896f4c4f1127dc2b99; rit=OPENING, location=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:26,889 DEBUG [PostOpenDeployTasks:8e2537011449fca6295a357cbde76e6c] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=6, pid=37, ppid=34, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=8e2537011449fca6295a357cbde76e6c; rit=OPENING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:26,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=6, pid=38, ppid=34, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=0dfb3ebb8c759d632ec793ac0387ec95; rit=OPENING, location=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:26,891 DEBUG [PEWorker-7] assignment.RegionTransitionProcedure(354): Finishing pid=36, ppid=34, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=28b5f974b61112896f4c4f1127dc2b99; rit=OPENING, location=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:26,891 DEBUG [PostOpenDeployTasks:8e2537011449fca6295a357cbde76e6c] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c. 2018-06-21 23:23:26,891 DEBUG [PostOpenDeployTasks:0dfb3ebb8c759d632ec793ac0387ec95] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95. 2018-06-21 23:23:26,891 INFO [PEWorker-7] assignment.RegionStateStore(198): pid=36 updating hbase:meta row=28b5f974b61112896f4c4f1127dc2b99, regionState=OPEN, openSeqNum=6, regionLocation=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:26,891 DEBUG [PEWorker-1] assignment.RegionTransitionProcedure(354): Finishing pid=38, ppid=34, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=0dfb3ebb8c759d632ec793ac0387ec95; rit=OPENING, location=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:26,891 DEBUG [PEWorker-9] assignment.RegionTransitionProcedure(354): Finishing pid=37, ppid=34, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=8e2537011449fca6295a357cbde76e6c; rit=OPENING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:26,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=6, pid=39, ppid=34, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=794028a2b8fc0a516890763306629d4c; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:26,891 DEBUG [PostOpenDeployTasks:28b5f974b61112896f4c4f1127dc2b99] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,mmm,1529623399386.28b5f974b61112896f4c4f1127dc2b99. 2018-06-21 23:23:26,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=6, pid=35, ppid=34, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=573e96916daa12dfc083e39a253dc3a2; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:26,893 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] handler.OpenRegionHandler(128): Opened multi_test_table,mmm,1529623399386.28b5f974b61112896f4c4f1127dc2b99. on asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:26,892 DEBUG [PEWorker-8] assignment.RegionTransitionProcedure(354): Finishing pid=39, ppid=34, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=794028a2b8fc0a516890763306629d4c; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:26,892 INFO [PEWorker-1] assignment.RegionStateStore(198): pid=38 updating hbase:meta row=0dfb3ebb8c759d632ec793ac0387ec95, regionState=OPEN, openSeqNum=6, regionLocation=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:26,893 INFO [PEWorker-8] assignment.RegionStateStore(198): pid=39 updating hbase:meta row=794028a2b8fc0a516890763306629d4c, regionState=OPEN, openSeqNum=6, regionLocation=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:26,892 INFO [PEWorker-9] assignment.RegionStateStore(198): pid=37 updating hbase:meta row=8e2537011449fca6295a357cbde76e6c, regionState=OPEN, openSeqNum=6, regionLocation=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:26,891 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] handler.OpenRegionHandler(128): Opened multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95. on asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:26,891 DEBUG [RS_OPEN_REGION-master/asf911:0-0] handler.OpenRegionHandler(128): Opened multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c. on asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:26,893 DEBUG [PostOpenDeployTasks:794028a2b8fc0a516890763306629d4c] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,zzz,1529623399386.794028a2b8fc0a516890763306629d4c. 2018-06-21 23:23:26,893 DEBUG [PEWorker-2] assignment.RegionTransitionProcedure(354): Finishing pid=35, ppid=34, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=573e96916daa12dfc083e39a253dc3a2; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:26,894 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] handler.OpenRegionHandler(128): Opened multi_test_table,zzz,1529623399386.794028a2b8fc0a516890763306629d4c. on asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:26,893 DEBUG [PostOpenDeployTasks:573e96916daa12dfc083e39a253dc3a2] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,jjj,1529623399386.573e96916daa12dfc083e39a253dc3a2. 2018-06-21 23:23:26,894 INFO [PEWorker-2] assignment.RegionStateStore(198): pid=35 updating hbase:meta row=573e96916daa12dfc083e39a253dc3a2, regionState=OPEN, openSeqNum=6, regionLocation=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:26,895 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] handler.OpenRegionHandler(128): Opened multi_test_table,jjj,1529623399386.573e96916daa12dfc083e39a253dc3a2. on asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:26,976 INFO [PEWorker-7] procedure2.ProcedureExecutor(1266): Finished pid=36, ppid=34, state=SUCCESS; AssignProcedure table=multi_test_table, region=28b5f974b61112896f4c4f1127dc2b99 in 713msec 2018-06-21 23:23:26,976 INFO [PEWorker-9] procedure2.ProcedureExecutor(1266): Finished pid=37, ppid=34, state=SUCCESS; AssignProcedure table=multi_test_table, region=8e2537011449fca6295a357cbde76e6c in 714msec 2018-06-21 23:23:26,976 INFO [PEWorker-8] procedure2.ProcedureExecutor(1266): Finished pid=39, ppid=34, state=SUCCESS; AssignProcedure table=multi_test_table, region=794028a2b8fc0a516890763306629d4c in 714msec 2018-06-21 23:23:26,976 INFO [PEWorker-1] procedure2.ProcedureExecutor(1266): Finished pid=38, ppid=34, state=SUCCESS; AssignProcedure table=multi_test_table, region=0dfb3ebb8c759d632ec793ac0387ec95 in 714msec 2018-06-21 23:23:27,038 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-06-21 23:23:27,138 INFO [PEWorker-2] procedure2.ProcedureExecutor(1635): Finished subprocedure(s) of pid=34, state=RUNNABLE:SERVER_CRASH_HANDLE_RIT2; ServerCrashProcedure server=asf911.gq1.ygridcore.net,59970,1529623390066, splitWal=true, meta=false; resume parent processing. 2018-06-21 23:23:27,139 INFO [PEWorker-2] procedure2.ProcedureExecutor(1266): Finished pid=35, ppid=34, state=SUCCESS; AssignProcedure table=multi_test_table, region=573e96916daa12dfc083e39a253dc3a2 in 714msec 2018-06-21 23:23:27,255 INFO [PEWorker-13] procedure2.ProcedureExecutor(1266): Finished pid=34, state=SUCCESS; ServerCrashProcedure server=asf911.gq1.ygridcore.net,59970,1529623390066, splitWal=true, meta=false in 1.6980sec 2018-06-21 23:23:27,550 INFO [Time-limited test] client.TestMultiParallel(755): Validating data on multi_test_table;hconnection-0x3389a216 2018-06-21 23:23:27,622 INFO [Time-limited test] client.TestMultiParallel(792): Validating data on multi_test_table;hconnection-0x3389a216 successfully! 2018-06-21 23:23:27,690 INFO [Time-limited test] hbase.ResourceChecker(172): after: client.TestMultiParallel#testBatchWithPut Thread=639 (was 628) Potentially hanging thread: RS_LOG_REPLAY_OPS-regionserver/asf911:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t43 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-shared-pool17-t18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t37 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_714311547_23 at /127.0.0.1:56763 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-9-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t26 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-shared-pool17-t10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-9-14 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-shared-pool17-t11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t51 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t34 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-9-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_267595925_23 at /127.0.0.1:44735 [Waiting for operation #6] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-shared-pool17-t1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t22 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t30 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t29 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-shared-pool17-t20 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-9-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t20 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t45 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-shared-pool17-t9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-shared-pool17-t3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_414825945_23 at /127.0.0.1:44755 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t23 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t33 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-shared-pool17-t15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1631786999_23 at /127.0.0.1:60380 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_714311547_23 at /127.0.0.1:44834 [Waiting for operation #10] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-6-4 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t44 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-shared-pool17-t17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t39 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-shared-pool17-t7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-shared-pool17-t13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-shared-pool17-t12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-5-4 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t41 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t40 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t36 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t32 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-9-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-shared-pool17-t16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-1-9 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t28 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-3-4 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t31 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-shared-pool17-t14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t42 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t48 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-shared-pool17-t4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t47 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_400506704_23 at /127.0.0.1:56725 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t19 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t46 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-shared-pool17-t19 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_400506704_23 at /127.0.0.1:44736 [Waiting for operation #6] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-9-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-shared-pool17-t8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t50 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-4-5 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t21 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t27 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1631786999_23 at /127.0.0.1:32937 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_400506704_23 at /127.0.0.1:32935 [Waiting for operation #7] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t49 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t35 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t24 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-shared-pool17-t5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-shared-pool17-t2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-9-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-shared-pool17-t6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0x3389a216-metaLookup-shared--pool18-t38 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) - Thread LEAK? -, OpenFileDescriptor=2437 (was 2444), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=601 (was 523) - SystemLoadAverage LEAK? -, ProcessCount=269 (was 269), AvailableMemoryMB=13470 (was 13509) 2018-06-21 23:23:27,691 WARN [Time-limited test] hbase.ResourceChecker(135): Thread=639 is superior to 500 2018-06-21 23:23:27,691 WARN [Time-limited test] hbase.ResourceChecker(135): OpenFileDescriptor=2437 is superior to 1024 2018-06-21 23:23:27,734 INFO [Time-limited test] hbase.ResourceChecker(148): before: client.TestMultiParallel#testFlushCommitsWithAbort Thread=639, OpenFileDescriptor=2437, MaxFileDescriptor=60000, SystemLoadAverage=601, ProcessCount=269, AvailableMemoryMB=13466 2018-06-21 23:23:27,734 WARN [Time-limited test] hbase.ResourceChecker(135): Thread=639 is superior to 500 2018-06-21 23:23:27,734 WARN [Time-limited test] hbase.ResourceChecker(135): OpenFileDescriptor=2437 is superior to 1024 2018-06-21 23:23:27,734 INFO [Time-limited test] client.TestMultiParallel(117): before 2018-06-21 23:23:27,734 INFO [Time-limited test] hbase.LocalHBaseCluster(270): Not alive RS:0;asf911:59970 2018-06-21 23:23:27,776 INFO [Time-limited test] client.ConnectionUtils(122): regionserver/asf911:0 server-side Connection retries=45 2018-06-21 23:23:27,777 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=50, handlerCount=5 2018-06-21 23:23:27,777 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated priority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=60, handlerCount=6 2018-06-21 23:23:27,777 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2018-06-21 23:23:27,777 INFO [Time-limited test] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-06-21 23:23:27,777 INFO [Time-limited test] io.ByteBufferPool(83): Created with bufferSize=64 KB and maxPoolSize=320 B 2018-06-21 23:23:27,778 INFO [Time-limited test] ipc.NettyRpcServer(109): Bind to /67.195.81.155:34033 2018-06-21 23:23:27,779 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=5, currentSize=750.81 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=750.81 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:27,780 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=5, currentSize=750.81 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=750.81 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:27,782 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-06-21 23:23:27,784 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-06-21 23:23:27,786 INFO [Time-limited test] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:34033 connecting to ZooKeeper ensemble=localhost:58989 2018-06-21 23:23:27,801 DEBUG [Time-limited test] zookeeper.ZKUtil(355): regionserver:340330x0, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-06-21 23:23:27,802 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:340330x0, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-06-21 23:23:27,805 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(543): regionserver:34033-0x16424a89dd4000f connected 2018-06-21 23:23:27,809 DEBUG [Time-limited test] zookeeper.ZKUtil(355): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2018-06-21 23:23:27,813 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=5 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34033 2018-06-21 23:23:27,818 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=6 with threadPrefix=priority.FPBQ.Fifo, numCallQueues=1, port=34033 2018-06-21 23:23:27,818 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34033 2018-06-21 23:23:27,832 INFO [RS:5;asf911:34033] regionserver.HRegionServer(874): ClusterId : 2ed7179d-84c9-432a-b6b5-556343aa7690 2018-06-21 23:23:27,834 DEBUG [RS:5;asf911:34033] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initializing 2018-06-21 23:23:27,844 DEBUG [RS:5;asf911:34033] procedure.RegionServerProcedureManagerHost(47): Procedure flush-table-proc initialized 2018-06-21 23:23:27,844 DEBUG [RS:5;asf911:34033] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initializing 2018-06-21 23:23:27,851 DEBUG [RS:5;asf911:34033] procedure.RegionServerProcedureManagerHost(47): Procedure online-snapshot initialized 2018-06-21 23:23:27,852 INFO [RS:5;asf911:34033] zookeeper.ReadOnlyZKClient(139): Connect 0x0160fc1e to localhost:58989 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-06-21 23:23:27,869 DEBUG [RS:5;asf911:34033] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@55f68ea, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-06-21 23:23:27,870 DEBUG [RS:5;asf911:34033] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@77d18667, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-06-21 23:23:27,870 DEBUG [RS:5;asf911:34033] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:5;asf911:34033 2018-06-21 23:23:27,870 INFO [RS:5;asf911:34033] regionserver.RegionServerCoprocessorHost(67): System coprocessor loading is enabled 2018-06-21 23:23:27,870 INFO [RS:5;asf911:34033] regionserver.RegionServerCoprocessorHost(68): Table coprocessor loading is enabled 2018-06-21 23:23:27,871 INFO [RS:5;asf911:34033] regionserver.HRegionServer(2600): reportForDuty to master=asf911.gq1.ygridcore.net,55146,1529623388534 with port=34033, startcode=1529623407773 2018-06-21 23:23:27,874 INFO [RS-EventLoopGroup-1-10] ipc.ServerRpcConnection(528): Auth successful for jenkins.hfs.5 (auth:SIMPLE) 2018-06-21 23:23:27,874 INFO [RS-EventLoopGroup-1-10] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 57607 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:27,875 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=55146] master.ServerManager(439): Registering regionserver=asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:27,876 DEBUG [RS:5;asf911:34033] regionserver.HRegionServer(1505): Config from master: hbase.rootdir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba 2018-06-21 23:23:27,878 DEBUG [RS:5;asf911:34033] regionserver.HRegionServer(1505): Config from master: fs.defaultFS=hdfs://localhost:56139 2018-06-21 23:23:27,878 DEBUG [RS:5;asf911:34033] regionserver.HRegionServer(1505): Config from master: hbase.master.info.port=-1 2018-06-21 23:23:27,883 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:27,883 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:27,883 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:27,883 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:27,883 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:27,884 DEBUG [RS:5;asf911:34033] zookeeper.ZKUtil(355): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:27,884 WARN [RS:5;asf911:34033] hbase.ZNodeClearer(63): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-06-21 23:23:27,885 INFO [RS:5;asf911:34033] wal.WALFactory(137): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-06-21 23:23:27,885 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:27,885 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:27,885 INFO [RegionServerTracker-0] master.RegionServerTracker(170): RegionServer ephemeral node created, adding [asf911.gq1.ygridcore.net,34033,1529623407773] 2018-06-21 23:23:27,885 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:27,885 DEBUG [RS:5;asf911:34033] regionserver.HRegionServer(1811): logDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:27,885 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:27,885 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:27,885 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:27,886 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:27,887 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:27,887 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:27,887 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:27,889 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:27,889 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:27,889 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:27,889 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:27,891 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:27,891 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:27,892 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:27,892 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:27,892 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:27,892 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:27,892 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:27,893 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:27,892 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:27,893 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:27,893 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:27,893 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:27,895 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:27,893 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:27,895 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:27,895 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:27,897 DEBUG [RS:5;asf911:34033] zookeeper.ZKUtil(355): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:27,898 DEBUG [RS:5;asf911:34033] zookeeper.ZKUtil(355): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:27,898 DEBUG [RS:5;asf911:34033] zookeeper.ZKUtil(355): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:27,899 DEBUG [RS:5;asf911:34033] zookeeper.ZKUtil(355): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:27,899 DEBUG [RS:5;asf911:34033] zookeeper.ZKUtil(355): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:27,900 DEBUG [RS:5;asf911:34033] zookeeper.ZKUtil(355): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:27,901 DEBUG [RS:5;asf911:34033] regionserver.Replication(132): Replication stats-in-log period=300 seconds 2018-06-21 23:23:27,901 INFO [RS:5;asf911:34033] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-06-21 23:23:27,904 INFO [RS:5;asf911:34033] regionserver.MemStoreFlusher(133): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-06-21 23:23:27,904 INFO [RS:5;asf911:34033] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/second, lower bound 10.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2018-06-21 23:23:27,904 INFO [RS:5;asf911:34033] regionserver.HRegionServer$CompactionChecker(1705): CompactionChecker runs every PT1S 2018-06-21 23:23:27,908 DEBUG [RS:5;asf911:34033] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:27,909 DEBUG [RS:5;asf911:34033] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-regionserver/asf911:0, corePoolSize=1, maxPoolSize=1 2018-06-21 23:23:27,909 DEBUG [RS:5;asf911:34033] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:27,909 DEBUG [RS:5;asf911:34033] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:27,909 DEBUG [RS:5;asf911:34033] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-regionserver/asf911:0, corePoolSize=1, maxPoolSize=1 2018-06-21 23:23:27,910 DEBUG [RS:5;asf911:34033] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/asf911:0, corePoolSize=2, maxPoolSize=2 2018-06-21 23:23:27,910 DEBUG [RS:5;asf911:34033] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/asf911:0, corePoolSize=10, maxPoolSize=10 2018-06-21 23:23:27,910 DEBUG [RS:5;asf911:34033] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:27,911 DEBUG [RS:5;asf911:34033] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-regionserver/asf911:0, corePoolSize=2, maxPoolSize=2 2018-06-21 23:23:27,938 INFO [RS:5;asf911:34033] regionserver.HeapMemoryManager(210): Starting, tuneOn=false 2018-06-21 23:23:27,938 INFO [SplitLogWorker-asf911:34033] regionserver.SplitLogWorker(136): SplitLogWorker asf911.gq1.ygridcore.net,34033,1529623407773 starting 2018-06-21 23:23:27,960 INFO [RS:5;asf911:34033] regionserver.HRegionServer(1546): Serving as asf911.gq1.ygridcore.net,34033,1529623407773, RpcServer on asf911.gq1.ygridcore.net/67.195.81.155:34033, sessionid=0x16424a89dd4000f 2018-06-21 23:23:27,960 DEBUG [RS:5;asf911:34033] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc starting 2018-06-21 23:23:27,960 INFO [Time-limited test] hbase.HBaseTestingUtility(3080): Started new server=Thread[RS:5;asf911:34033,5,FailOnTimeoutGroup] 2018-06-21 23:23:27,960 DEBUG [RS:5;asf911:34033] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:27,960 DEBUG [RS:5;asf911:34033] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'asf911.gq1.ygridcore.net,34033,1529623407773' 2018-06-21 23:23:27,961 DEBUG [RS:5;asf911:34033] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-06-21 23:23:27,961 DEBUG [RS:5;asf911:34033] procedure.ZKProcedureMemberRpcs(155): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-06-21 23:23:27,962 DEBUG [RS:5;asf911:34033] procedure.RegionServerProcedureManagerHost(55): Procedure flush-table-proc started 2018-06-21 23:23:27,962 DEBUG [RS:5;asf911:34033] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot starting 2018-06-21 23:23:27,962 DEBUG [RS:5;asf911:34033] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:27,962 DEBUG [RS:5;asf911:34033] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'asf911.gq1.ygridcore.net,34033,1529623407773' 2018-06-21 23:23:27,962 DEBUG [RS:5;asf911:34033] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-06-21 23:23:27,963 INFO [Time-limited test] master.HMaster(1534): Balancer plans size is 5, the balance interval is 60000 ms, and the max number regions in transition is 28 2018-06-21 23:23:27,963 DEBUG [RS:5;asf911:34033] procedure.ZKProcedureMemberRpcs(155): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-06-21 23:23:27,963 INFO [Time-limited test] master.HMaster(1539): balance hri=8e2537011449fca6295a357cbde76e6c, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:27,963 DEBUG [RS:5;asf911:34033] procedure.RegionServerProcedureManagerHost(55): Procedure online-snapshot started 2018-06-21 23:23:27,964 INFO [RS:5;asf911:34033] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-06-21 23:23:27,964 INFO [RS:5;asf911:34033] quotas.RegionServerSpaceQuotaManager(84): Quota support disabled, not starting space quota manager. 2018-06-21 23:23:28,138 DEBUG [Time-limited test] procedure2.ProcedureExecutor(887): Stored pid=40, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=8e2537011449fca6295a357cbde76e6c, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:28,141 INFO [PEWorker-15] procedure.MasterProcedureScheduler(696): pid=40, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=8e2537011449fca6295a357cbde76e6c, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,58235,1529623390382 checking lock on 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:28,141 INFO [Time-limited test] master.HMaster(1539): balance hri=fee67ae5d4f2b2be35fa516877a8d173, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:28,143 INFO [PEWorker-15] procedure2.ProcedureExecutor(1516): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=multi_test_table, region=8e2537011449fca6295a357cbde76e6c, server=asf911.gq1.ygridcore.net,55146,1529623388534}] 2018-06-21 23:23:28,238 DEBUG [Time-limited test] procedure2.ProcedureExecutor(887): Stored pid=42, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=fee67ae5d4f2b2be35fa516877a8d173, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:28,239 INFO [Time-limited test] master.HMaster(1539): balance hri=04d71925444c7c823938ba3d9ad0504d, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:28,241 INFO [PEWorker-3] procedure.MasterProcedureScheduler(696): pid=42, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=fee67ae5d4f2b2be35fa516877a8d173, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,40767,1529623390148 checking lock on fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:28,241 INFO [PEWorker-3] procedure2.ProcedureExecutor(1516): Initialized subprocedures=[{pid=43, ppid=42, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=multi_test_table, region=fee67ae5d4f2b2be35fa516877a8d173, server=asf911.gq1.ygridcore.net,55146,1529623388534}] 2018-06-21 23:23:28,320 INFO [PEWorker-15] procedure.MasterProcedureScheduler(696): pid=41, ppid=40, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=multi_test_table, region=8e2537011449fca6295a357cbde76e6c, server=asf911.gq1.ygridcore.net,55146,1529623388534 checking lock on 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:28,321 INFO [PEWorker-15] assignment.RegionStateStore(198): pid=41 updating hbase:meta row=8e2537011449fca6295a357cbde76e6c, regionState=CLOSING, regionLocation=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:28,324 INFO [PEWorker-15] assignment.RegionTransitionProcedure(241): Dispatch pid=41, ppid=40, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=multi_test_table, region=8e2537011449fca6295a357cbde76e6c, server=asf911.gq1.ygridcore.net,55146,1529623388534; rit=CLOSING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:28,439 INFO [PEWorker-14] procedure.MasterProcedureScheduler(696): pid=43, ppid=42, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=multi_test_table, region=fee67ae5d4f2b2be35fa516877a8d173, server=asf911.gq1.ygridcore.net,55146,1529623388534 checking lock on fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:28,439 INFO [PEWorker-14] assignment.RegionStateStore(198): pid=43 updating hbase:meta row=fee67ae5d4f2b2be35fa516877a8d173, regionState=CLOSING, regionLocation=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:28,443 INFO [PEWorker-14] assignment.RegionTransitionProcedure(241): Dispatch pid=43, ppid=42, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=multi_test_table, region=fee67ae5d4f2b2be35fa516877a8d173, server=asf911.gq1.ygridcore.net,55146,1529623388534; rit=CLOSING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:28,474 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(119): Fallback to compat rpc execution for serverName=asf911.gq1.ygridcore.net,55146,1529623388534 version=0 2018-06-21 23:23:28,479 INFO [RSProcedureDispatcher-pool3-t17] regionserver.RSRpcServices(1604): Close 8e2537011449fca6295a357cbde76e6c, moving to asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:28,479 INFO [RSProcedureDispatcher-pool3-t18] regionserver.RSRpcServices(1604): Close fee67ae5d4f2b2be35fa516877a8d173, moving to asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:28,482 DEBUG [RS_CLOSE_REGION-master/asf911:0-0] regionserver.HRegion(1520): Closing 8e2537011449fca6295a357cbde76e6c, disabling compactions & flushes 2018-06-21 23:23:28,482 DEBUG [RS_CLOSE_REGION-master/asf911:0-0] regionserver.HRegion(1560): Updates disabled for region multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c. 2018-06-21 23:23:28,484 DEBUG [RS_CLOSE_REGION-master/asf911:0-1] regionserver.HRegion(1520): Closing fee67ae5d4f2b2be35fa516877a8d173, disabling compactions & flushes 2018-06-21 23:23:28,484 INFO [RS_CLOSE_REGION-master/asf911:0-0] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=546 B heapSize=904 B 2018-06-21 23:23:28,484 DEBUG [RS_CLOSE_REGION-master/asf911:0-1] regionserver.HRegion(1560): Updates disabled for region multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173. 2018-06-21 23:23:28,484 INFO [RS_CLOSE_REGION-master/asf911:0-1] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=1.60 KB heapSize=3.28 KB 2018-06-21 23:23:28,498 DEBUG [Time-limited test] procedure2.ProcedureExecutor(887): Stored pid=44, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=04d71925444c7c823938ba3d9ad0504d, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:28,500 INFO [PEWorker-16] procedure.MasterProcedureScheduler(696): pid=44, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=04d71925444c7c823938ba3d9ad0504d, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,52507,1529623390228 checking lock on 04d71925444c7c823938ba3d9ad0504d 2018-06-21 23:23:28,501 INFO [PEWorker-16] procedure2.ProcedureExecutor(1516): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=multi_test_table, region=04d71925444c7c823938ba3d9ad0504d, server=asf911.gq1.ygridcore.net,55146,1529623388534}] 2018-06-21 23:23:28,548 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741876_1052{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|FINALIZED]]} size 0 2018-06-21 23:23:28,549 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741877_1053{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|FINALIZED]]} size 0 2018-06-21 23:23:28,549 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741876_1052{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|FINALIZED], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|FINALIZED]]} size 0 2018-06-21 23:23:28,549 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741877_1053{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|FINALIZED], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:28,549 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741877_1053 size 5232 2018-06-21 23:23:28,574 INFO [RS_CLOSE_REGION-master/asf911:0-0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=546 B at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/.tmp/family/18485bb517bd4fdab3bd9cad5675a69a 2018-06-21 23:23:28,575 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741876_1052 size 5232 2018-06-21 23:23:28,575 INFO [RS_CLOSE_REGION-master/asf911:0-1] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=1.60 KB at sequenceid=7 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/.tmp/family/7faefec2067642c89a50ae799bba98ba 2018-06-21 23:23:28,590 DEBUG [RS_CLOSE_REGION-master/asf911:0-0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/.tmp/family/18485bb517bd4fdab3bd9cad5675a69a as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/family/18485bb517bd4fdab3bd9cad5675a69a 2018-06-21 23:23:28,590 DEBUG [RS_CLOSE_REGION-master/asf911:0-1] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/.tmp/family/7faefec2067642c89a50ae799bba98ba as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/family/7faefec2067642c89a50ae799bba98ba 2018-06-21 23:23:28,627 INFO [RS_CLOSE_REGION-master/asf911:0-1] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/family/7faefec2067642c89a50ae799bba98ba, entries=10, sequenceid=7, filesize=5.1 K 2018-06-21 23:23:28,627 INFO [RS_CLOSE_REGION-master/asf911:0-0] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/family/18485bb517bd4fdab3bd9cad5675a69a, entries=10, sequenceid=9, filesize=5.1 K 2018-06-21 23:23:28,632 INFO [RS_CLOSE_REGION-master/asf911:0-0] regionserver.HRegion(2769): Finished flush of dataSize ~546 B/546, heapSize ~1.30 KB/1336, currentSize=0 B/0 for 8e2537011449fca6295a357cbde76e6c in 147ms, sequenceid=9, compaction requested=false 2018-06-21 23:23:28,632 INFO [RS_CLOSE_REGION-master/asf911:0-1] regionserver.HRegion(2769): Finished flush of dataSize ~1.60 KB/1638, heapSize ~3.49 KB/3576, currentSize=0 B/0 for fee67ae5d4f2b2be35fa516877a8d173 in 147ms, sequenceid=7, compaction requested=false 2018-06-21 23:23:28,645 INFO [PEWorker-16] procedure.MasterProcedureScheduler(696): pid=45, ppid=44, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=multi_test_table, region=04d71925444c7c823938ba3d9ad0504d, server=asf911.gq1.ygridcore.net,55146,1529623388534 checking lock on 04d71925444c7c823938ba3d9ad0504d 2018-06-21 23:23:28,645 DEBUG [RS_CLOSE_REGION-master/asf911:0-1] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/recovered.edits/10.seqid, newMaxSeqId=10, maxSeqId=1 2018-06-21 23:23:28,645 DEBUG [RS_CLOSE_REGION-master/asf911:0-0] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/recovered.edits/12.seqid, newMaxSeqId=12, maxSeqId=5 2018-06-21 23:23:28,645 INFO [PEWorker-16] assignment.RegionStateStore(198): pid=45 updating hbase:meta row=04d71925444c7c823938ba3d9ad0504d, regionState=CLOSING, regionLocation=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:28,648 INFO [RS_CLOSE_REGION-master/asf911:0-0] regionserver.HRegion(1674): Closed multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c. 2018-06-21 23:23:28,649 INFO [RS_CLOSE_REGION-master/asf911:0-0] regionserver.HRegionServer(3426): Adding 8e2537011449fca6295a357cbde76e6c move to asf911.gq1.ygridcore.net,58235,1529623390382 record at close sequenceid=9 2018-06-21 23:23:28,649 INFO [RS_CLOSE_REGION-master/asf911:0-1] regionserver.HRegion(1674): Closed multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173. 2018-06-21 23:23:28,649 INFO [RS_CLOSE_REGION-master/asf911:0-1] regionserver.HRegionServer(3426): Adding fee67ae5d4f2b2be35fa516877a8d173 move to asf911.gq1.ygridcore.net,40767,1529623390148 record at close sequenceid=7 2018-06-21 23:23:28,650 DEBUG [RS_CLOSE_REGION-master/asf911:0-0] assignment.RegionTransitionProcedure(264): Received report CLOSED seqId=-1, pid=41, ppid=40, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=multi_test_table, region=8e2537011449fca6295a357cbde76e6c, server=asf911.gq1.ygridcore.net,55146,1529623388534; rit=CLOSING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:28,650 DEBUG [RS_CLOSE_REGION-master/asf911:0-1] assignment.RegionTransitionProcedure(264): Received report CLOSED seqId=-1, pid=43, ppid=42, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=multi_test_table, region=fee67ae5d4f2b2be35fa516877a8d173, server=asf911.gq1.ygridcore.net,55146,1529623388534; rit=CLOSING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:28,651 DEBUG [RS_CLOSE_REGION-master/asf911:0-0] handler.CloseRegionHandler(124): Closed multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c. 2018-06-21 23:23:28,651 DEBUG [RS_CLOSE_REGION-master/asf911:0-1] handler.CloseRegionHandler(124): Closed multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173. 2018-06-21 23:23:28,651 DEBUG [PEWorker-11] assignment.RegionTransitionProcedure(354): Finishing pid=43, ppid=42, state=RUNNABLE:REGION_TRANSITION_FINISH; UnassignProcedure table=multi_test_table, region=fee67ae5d4f2b2be35fa516877a8d173, server=asf911.gq1.ygridcore.net,55146,1529623388534; rit=CLOSING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:28,652 DEBUG [PEWorker-10] assignment.RegionTransitionProcedure(354): Finishing pid=41, ppid=40, state=RUNNABLE:REGION_TRANSITION_FINISH; UnassignProcedure table=multi_test_table, region=8e2537011449fca6295a357cbde76e6c, server=asf911.gq1.ygridcore.net,55146,1529623388534; rit=CLOSING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:28,653 INFO [PEWorker-11] assignment.RegionStateStore(198): pid=43 updating hbase:meta row=fee67ae5d4f2b2be35fa516877a8d173, regionState=CLOSED 2018-06-21 23:23:28,653 INFO [PEWorker-16] assignment.RegionTransitionProcedure(241): Dispatch pid=45, ppid=44, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=multi_test_table, region=04d71925444c7c823938ba3d9ad0504d, server=asf911.gq1.ygridcore.net,55146,1529623388534; rit=CLOSING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:28,653 INFO [PEWorker-10] assignment.RegionStateStore(198): pid=41 updating hbase:meta row=8e2537011449fca6295a357cbde76e6c, regionState=CLOSED 2018-06-21 23:23:28,790 INFO [PEWorker-11] procedure2.ProcedureExecutor(1635): Finished subprocedure(s) of pid=42, state=RUNNABLE:MOVE_REGION_ASSIGN; MoveRegionProcedure hri=fee67ae5d4f2b2be35fa516877a8d173, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,40767,1529623390148; resume parent processing. 2018-06-21 23:23:28,790 INFO [PEWorker-10] procedure2.ProcedureExecutor(1635): Finished subprocedure(s) of pid=40, state=RUNNABLE:MOVE_REGION_ASSIGN; MoveRegionProcedure hri=8e2537011449fca6295a357cbde76e6c, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,58235,1529623390382; resume parent processing. 2018-06-21 23:23:28,790 INFO [PEWorker-4] procedure2.ProcedureExecutor(1516): Initialized subprocedures=[{pid=46, ppid=42, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=fee67ae5d4f2b2be35fa516877a8d173, target=asf911.gq1.ygridcore.net,40767,1529623390148}] 2018-06-21 23:23:28,790 INFO [PEWorker-7] procedure2.ProcedureExecutor(1516): Initialized subprocedures=[{pid=47, ppid=40, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=8e2537011449fca6295a357cbde76e6c, target=asf911.gq1.ygridcore.net,58235,1529623390382}] 2018-06-21 23:23:28,790 INFO [PEWorker-10] procedure2.ProcedureExecutor(1266): Finished pid=41, ppid=40, state=SUCCESS; UnassignProcedure table=multi_test_table, region=8e2537011449fca6295a357cbde76e6c, server=asf911.gq1.ygridcore.net,55146,1529623388534 in 513msec 2018-06-21 23:23:28,790 INFO [PEWorker-11] procedure2.ProcedureExecutor(1266): Finished pid=43, ppid=42, state=SUCCESS; UnassignProcedure table=multi_test_table, region=fee67ae5d4f2b2be35fa516877a8d173, server=asf911.gq1.ygridcore.net,55146,1529623388534 in 414msec 2018-06-21 23:23:28,803 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(119): Fallback to compat rpc execution for serverName=asf911.gq1.ygridcore.net,55146,1529623388534 version=0 2018-06-21 23:23:28,804 INFO [RSProcedureDispatcher-pool3-t20] regionserver.RSRpcServices(1604): Close 04d71925444c7c823938ba3d9ad0504d, moving to asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:28,807 DEBUG [RS_CLOSE_REGION-master/asf911:0-2] regionserver.HRegion(1520): Closing 04d71925444c7c823938ba3d9ad0504d, disabling compactions & flushes 2018-06-21 23:23:28,807 DEBUG [RS_CLOSE_REGION-master/asf911:0-2] regionserver.HRegion(1560): Updates disabled for region multi_test_table,iii,1529623399386.04d71925444c7c823938ba3d9ad0504d. 2018-06-21 23:23:28,807 INFO [RS_CLOSE_REGION-master/asf911:0-2] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=1.60 KB heapSize=3.28 KB 2018-06-21 23:23:28,826 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741878_1054{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|FINALIZED]]} size 0 2018-06-21 23:23:28,827 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741878_1054{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|FINALIZED], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|FINALIZED]]} size 0 2018-06-21 23:23:28,827 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741878_1054{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|FINALIZED], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|FINALIZED], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|FINALIZED]]} size 0 2018-06-21 23:23:28,828 INFO [RS_CLOSE_REGION-master/asf911:0-2] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=1.60 KB at sequenceid=7 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/04d71925444c7c823938ba3d9ad0504d/.tmp/family/d7b2b05345e04cb986c20c56bcc17dfd 2018-06-21 23:23:28,885 DEBUG [RS_CLOSE_REGION-master/asf911:0-2] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/04d71925444c7c823938ba3d9ad0504d/.tmp/family/d7b2b05345e04cb986c20c56bcc17dfd as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/04d71925444c7c823938ba3d9ad0504d/family/d7b2b05345e04cb986c20c56bcc17dfd 2018-06-21 23:23:28,895 INFO [RS_CLOSE_REGION-master/asf911:0-2] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/04d71925444c7c823938ba3d9ad0504d/family/d7b2b05345e04cb986c20c56bcc17dfd, entries=10, sequenceid=7, filesize=5.1 K 2018-06-21 23:23:28,899 INFO [RS_CLOSE_REGION-master/asf911:0-2] regionserver.HRegion(2769): Finished flush of dataSize ~1.60 KB/1638, heapSize ~3.49 KB/3576, currentSize=0 B/0 for 04d71925444c7c823938ba3d9ad0504d in 92ms, sequenceid=7, compaction requested=false 2018-06-21 23:23:28,914 DEBUG [RS_CLOSE_REGION-master/asf911:0-2] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/04d71925444c7c823938ba3d9ad0504d/recovered.edits/10.seqid, newMaxSeqId=10, maxSeqId=1 2018-06-21 23:23:28,918 INFO [RS_CLOSE_REGION-master/asf911:0-2] regionserver.HRegion(1674): Closed multi_test_table,iii,1529623399386.04d71925444c7c823938ba3d9ad0504d. 2018-06-21 23:23:28,918 INFO [RS_CLOSE_REGION-master/asf911:0-2] regionserver.HRegionServer(3426): Adding 04d71925444c7c823938ba3d9ad0504d move to asf911.gq1.ygridcore.net,52507,1529623390228 record at close sequenceid=7 2018-06-21 23:23:28,919 DEBUG [RS_CLOSE_REGION-master/asf911:0-2] assignment.RegionTransitionProcedure(264): Received report CLOSED seqId=-1, pid=45, ppid=44, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=multi_test_table, region=04d71925444c7c823938ba3d9ad0504d, server=asf911.gq1.ygridcore.net,55146,1529623388534; rit=CLOSING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:28,919 DEBUG [RS_CLOSE_REGION-master/asf911:0-2] handler.CloseRegionHandler(124): Closed multi_test_table,iii,1529623399386.04d71925444c7c823938ba3d9ad0504d. 2018-06-21 23:23:28,920 DEBUG [PEWorker-9] assignment.RegionTransitionProcedure(354): Finishing pid=45, ppid=44, state=RUNNABLE:REGION_TRANSITION_FINISH; UnassignProcedure table=multi_test_table, region=04d71925444c7c823938ba3d9ad0504d, server=asf911.gq1.ygridcore.net,55146,1529623388534; rit=CLOSING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:28,920 INFO [PEWorker-9] assignment.RegionStateStore(198): pid=45 updating hbase:meta row=04d71925444c7c823938ba3d9ad0504d, regionState=CLOSED 2018-06-21 23:23:28,947 INFO [PEWorker-4] procedure.MasterProcedureScheduler(696): pid=47, ppid=40, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=8e2537011449fca6295a357cbde76e6c, target=asf911.gq1.ygridcore.net,58235,1529623390382 checking lock on 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:28,947 INFO [PEWorker-7] procedure.MasterProcedureScheduler(696): pid=46, ppid=42, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=fee67ae5d4f2b2be35fa516877a8d173, target=asf911.gq1.ygridcore.net,40767,1529623390148 checking lock on fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:28,949 INFO [PEWorker-7] assignment.AssignProcedure(218): Starting pid=46, ppid=42, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=fee67ae5d4f2b2be35fa516877a8d173, target=asf911.gq1.ygridcore.net,40767,1529623390148; rit=OFFLINE, location=asf911.gq1.ygridcore.net,40767,1529623390148; forceNewPlan=false, retain=false 2018-06-21 23:23:28,949 INFO [PEWorker-4] assignment.AssignProcedure(218): Starting pid=47, ppid=40, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=8e2537011449fca6295a357cbde76e6c, target=asf911.gq1.ygridcore.net,58235,1529623390382; rit=OFFLINE, location=asf911.gq1.ygridcore.net,58235,1529623390382; forceNewPlan=false, retain=false 2018-06-21 23:23:28,964 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@69ed135] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:33402 to delete [blk_1073741866_1042, blk_1073741867_1043, blk_1073741869_1045, blk_1073741870_1046] 2018-06-21 23:23:28,964 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@69ed135] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:57095 to delete [blk_1073741866_1042, blk_1073741868_1044] 2018-06-21 23:23:28,968 INFO [RS:5;asf911:34033] wal.AbstractFSWAL(424): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C34033%2C1529623407773, suffix=, logDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,34033,1529623407773, archiveDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/oldWALs 2018-06-21 23:23:28,980 DEBUG [RS-EventLoopGroup-10-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:53553,DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4,DISK] 2018-06-21 23:23:28,980 DEBUG [RS-EventLoopGroup-10-4] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42074,DS-85f0e371-3357-43c0-95ea-ca66d237b4a8,DISK] 2018-06-21 23:23:28,980 DEBUG [RS-EventLoopGroup-10-5] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33402,DS-189d75fd-2cba-4781-8f7e-7f1fecbda947,DISK] 2018-06-21 23:23:28,983 INFO [RS:5;asf911:34033] wal.AbstractFSWAL(679): New WAL /user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,34033,1529623407773/asf911.gq1.ygridcore.net%2C34033%2C1529623407773.1529623408969 2018-06-21 23:23:28,984 DEBUG [RS:5;asf911:34033] wal.AbstractFSWAL(769): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:53553,DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4,DISK], DatanodeInfoWithStorage[127.0.0.1:42074,DS-85f0e371-3357-43c0-95ea-ca66d237b4a8,DISK], DatanodeInfoWithStorage[127.0.0.1:33402,DS-189d75fd-2cba-4781-8f7e-7f1fecbda947,DISK]] 2018-06-21 23:23:29,099 INFO [master/asf911:0] balancer.BaseLoadBalancer(1497): Reassigned 2 regions. 2 retained the pre-restart assignment. 2018-06-21 23:23:29,100 INFO [PEWorker-6] assignment.RegionStateStore(198): pid=46 updating hbase:meta row=fee67ae5d4f2b2be35fa516877a8d173, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:29,100 INFO [PEWorker-2] assignment.RegionStateStore(198): pid=47 updating hbase:meta row=8e2537011449fca6295a357cbde76e6c, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:29,104 INFO [PEWorker-2] assignment.RegionTransitionProcedure(241): Dispatch pid=47, ppid=40, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=8e2537011449fca6295a357cbde76e6c, target=asf911.gq1.ygridcore.net,58235,1529623390382; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:29,104 INFO [PEWorker-6] assignment.RegionTransitionProcedure(241): Dispatch pid=46, ppid=42, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=fee67ae5d4f2b2be35fa516877a8d173, target=asf911.gq1.ygridcore.net,40767,1529623390148; rit=OPENING, location=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:29,107 INFO [PEWorker-9] procedure2.ProcedureExecutor(1635): Finished subprocedure(s) of pid=44, state=RUNNABLE:MOVE_REGION_ASSIGN; MoveRegionProcedure hri=04d71925444c7c823938ba3d9ad0504d, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,52507,1529623390228; resume parent processing. 2018-06-21 23:23:29,107 INFO [PEWorker-9] procedure2.ProcedureExecutor(1266): Finished pid=45, ppid=44, state=SUCCESS; UnassignProcedure table=multi_test_table, region=04d71925444c7c823938ba3d9ad0504d, server=asf911.gq1.ygridcore.net,55146,1529623388534 in 423msec 2018-06-21 23:23:29,107 INFO [PEWorker-13] procedure2.ProcedureExecutor(1516): Initialized subprocedures=[{pid=48, ppid=44, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=04d71925444c7c823938ba3d9ad0504d, target=asf911.gq1.ygridcore.net,52507,1529623390228}] 2018-06-21 23:23:29,256 INFO [RpcServer.priority.FPBQ.Fifo.handler=1,queue=0,port=40767] regionserver.RSRpcServices(1980): Open multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173. 2018-06-21 23:23:29,256 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58235] regionserver.RSRpcServices(1980): Open multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c. 2018-06-21 23:23:29,261 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7094): Opening region: {ENCODED => fee67ae5d4f2b2be35fa516877a8d173, NAME => 'multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173.', STARTKEY => 'kkk', ENDKEY => 'lll'} 2018-06-21 23:23:29,261 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:29,261 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7094): Opening region: {ENCODED => 8e2537011449fca6295a357cbde76e6c, NAME => 'multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c.', STARTKEY => 'rrr', ENDKEY => 'sss'} 2018-06-21 23:23:29,261 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(827): Instantiated multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:29,262 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7134): checking encryption for fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:29,262 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7139): checking classloading for fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:29,262 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:29,262 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(827): Instantiated multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:29,262 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7134): checking encryption for 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:29,262 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7139): checking classloading for 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:29,266 DEBUG [StoreOpener-fee67ae5d4f2b2be35fa516877a8d173-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/family 2018-06-21 23:23:29,266 DEBUG [StoreOpener-fee67ae5d4f2b2be35fa516877a8d173-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/family 2018-06-21 23:23:29,267 DEBUG [StoreOpener-8e2537011449fca6295a357cbde76e6c-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/family 2018-06-21 23:23:29,267 DEBUG [StoreOpener-8e2537011449fca6295a357cbde76e6c-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/family 2018-06-21 23:23:29,267 INFO [StoreOpener-fee67ae5d4f2b2be35fa516877a8d173-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=5, currentSize=750.81 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=750.81 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:29,268 INFO [StoreOpener-fee67ae5d4f2b2be35fa516877a8d173-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:29,268 INFO [StoreOpener-8e2537011449fca6295a357cbde76e6c-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=5, currentSize=750.81 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=750.81 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:29,268 INFO [StoreOpener-8e2537011449fca6295a357cbde76e6c-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:29,282 INFO [PEWorker-13] procedure.MasterProcedureScheduler(696): pid=48, ppid=44, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=04d71925444c7c823938ba3d9ad0504d, target=asf911.gq1.ygridcore.net,52507,1529623390228 checking lock on 04d71925444c7c823938ba3d9ad0504d 2018-06-21 23:23:29,284 DEBUG [StoreOpener-fee67ae5d4f2b2be35fa516877a8d173-1] regionserver.HStore(581): loaded hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/family/7faefec2067642c89a50ae799bba98ba 2018-06-21 23:23:29,284 INFO [PEWorker-13] assignment.AssignProcedure(218): Starting pid=48, ppid=44, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=04d71925444c7c823938ba3d9ad0504d, target=asf911.gq1.ygridcore.net,52507,1529623390228; rit=OFFLINE, location=asf911.gq1.ygridcore.net,52507,1529623390228; forceNewPlan=false, retain=false 2018-06-21 23:23:29,285 INFO [StoreOpener-fee67ae5d4f2b2be35fa516877a8d173-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:29,285 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(923): replaying wal for fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:29,285 DEBUG [StoreOpener-8e2537011449fca6295a357cbde76e6c-1] regionserver.HStore(581): loaded hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/family/18485bb517bd4fdab3bd9cad5675a69a 2018-06-21 23:23:29,288 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:29,288 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(931): stopping wal replay for fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:29,288 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(943): Cleaning up temporary data for fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:29,289 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(954): Cleaning up detritus for fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:29,292 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(976): writing seq id for fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:29,293 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(980): Opened fee67ae5d4f2b2be35fa516877a8d173; next sequenceid=11 2018-06-21 23:23:29,293 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(987): Running coprocessor post-open hooks for fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:29,293 DEBUG [StoreOpener-8e2537011449fca6295a357cbde76e6c-1] regionserver.HStore(581): loaded hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/family/7f5cd2b2372e4de08c448cfcd864eded 2018-06-21 23:23:29,294 INFO [StoreOpener-8e2537011449fca6295a357cbde76e6c-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:29,294 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(923): replaying wal for 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:29,296 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:29,296 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(931): stopping wal replay for 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:29,296 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(943): Cleaning up temporary data for 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:29,318 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(954): Cleaning up detritus for 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:29,318 INFO [PostOpenDeployTasks:fee67ae5d4f2b2be35fa516877a8d173] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173. 2018-06-21 23:23:29,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=11, pid=46, ppid=42, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=fee67ae5d4f2b2be35fa516877a8d173, target=asf911.gq1.ygridcore.net,40767,1529623390148; rit=OPENING, location=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:29,320 DEBUG [PEWorker-3] assignment.RegionTransitionProcedure(354): Finishing pid=46, ppid=42, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=fee67ae5d4f2b2be35fa516877a8d173, target=asf911.gq1.ygridcore.net,40767,1529623390148; rit=OPENING, location=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:29,320 INFO [PEWorker-3] assignment.RegionStateStore(198): pid=46 updating hbase:meta row=fee67ae5d4f2b2be35fa516877a8d173, regionState=OPEN, openSeqNum=11, regionLocation=asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:29,320 DEBUG [PostOpenDeployTasks:fee67ae5d4f2b2be35fa516877a8d173] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173. 2018-06-21 23:23:29,321 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] handler.OpenRegionHandler(128): Opened multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173. on asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:29,321 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(976): writing seq id for 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:29,322 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(980): Opened 8e2537011449fca6295a357cbde76e6c; next sequenceid=13 2018-06-21 23:23:29,322 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(987): Running coprocessor post-open hooks for 8e2537011449fca6295a357cbde76e6c 2018-06-21 23:23:29,324 INFO [PostOpenDeployTasks:8e2537011449fca6295a357cbde76e6c] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c. 2018-06-21 23:23:29,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=13, pid=47, ppid=40, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=8e2537011449fca6295a357cbde76e6c, target=asf911.gq1.ygridcore.net,58235,1529623390382; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:29,325 DEBUG [PEWorker-15] assignment.RegionTransitionProcedure(354): Finishing pid=47, ppid=40, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=8e2537011449fca6295a357cbde76e6c, target=asf911.gq1.ygridcore.net,58235,1529623390382; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:29,326 DEBUG [PostOpenDeployTasks:8e2537011449fca6295a357cbde76e6c] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c. 2018-06-21 23:23:29,326 INFO [PEWorker-15] assignment.RegionStateStore(198): pid=47 updating hbase:meta row=8e2537011449fca6295a357cbde76e6c, regionState=OPEN, openSeqNum=13, regionLocation=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:29,326 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] handler.OpenRegionHandler(128): Opened multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c. on asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:29,340 INFO [PEWorker-2] procedure2.ProcedureExecutor(1266): Finished pid=47, ppid=40, state=SUCCESS; AssignProcedure table=multi_test_table, region=8e2537011449fca6295a357cbde76e6c, target=asf911.gq1.ygridcore.net,58235,1529623390382 in 539msec 2018-06-21 23:23:29,435 INFO [master/asf911:0] balancer.BaseLoadBalancer(1497): Reassigned 1 regions. 1 retained the pre-restart assignment. 2018-06-21 23:23:29,435 INFO [PEWorker-14] assignment.RegionStateStore(198): pid=48 updating hbase:meta row=04d71925444c7c823938ba3d9ad0504d, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:29,439 INFO [PEWorker-14] assignment.RegionTransitionProcedure(241): Dispatch pid=48, ppid=44, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=04d71925444c7c823938ba3d9ad0504d, target=asf911.gq1.ygridcore.net,52507,1529623390228; rit=OPENING, location=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:29,591 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=52507] regionserver.RSRpcServices(1980): Open multi_test_table,iii,1529623399386.04d71925444c7c823938ba3d9ad0504d. 2018-06-21 23:23:29,596 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7094): Opening region: {ENCODED => 04d71925444c7c823938ba3d9ad0504d, NAME => 'multi_test_table,iii,1529623399386.04d71925444c7c823938ba3d9ad0504d.', STARTKEY => 'iii', ENDKEY => 'jjj'} 2018-06-21 23:23:29,597 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table 04d71925444c7c823938ba3d9ad0504d 2018-06-21 23:23:29,597 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(827): Instantiated multi_test_table,iii,1529623399386.04d71925444c7c823938ba3d9ad0504d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:29,597 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7134): checking encryption for 04d71925444c7c823938ba3d9ad0504d 2018-06-21 23:23:29,597 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7139): checking classloading for 04d71925444c7c823938ba3d9ad0504d 2018-06-21 23:23:29,601 DEBUG [StoreOpener-04d71925444c7c823938ba3d9ad0504d-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/04d71925444c7c823938ba3d9ad0504d/family 2018-06-21 23:23:29,601 DEBUG [StoreOpener-04d71925444c7c823938ba3d9ad0504d-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/04d71925444c7c823938ba3d9ad0504d/family 2018-06-21 23:23:29,602 INFO [StoreOpener-04d71925444c7c823938ba3d9ad0504d-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=5, currentSize=750.81 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=750.81 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:29,602 INFO [StoreOpener-04d71925444c7c823938ba3d9ad0504d-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:29,614 DEBUG [StoreOpener-04d71925444c7c823938ba3d9ad0504d-1] regionserver.HStore(581): loaded hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/04d71925444c7c823938ba3d9ad0504d/family/d7b2b05345e04cb986c20c56bcc17dfd 2018-06-21 23:23:29,614 INFO [StoreOpener-04d71925444c7c823938ba3d9ad0504d-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:29,614 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(923): replaying wal for 04d71925444c7c823938ba3d9ad0504d 2018-06-21 23:23:29,617 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/04d71925444c7c823938ba3d9ad0504d 2018-06-21 23:23:29,617 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(931): stopping wal replay for 04d71925444c7c823938ba3d9ad0504d 2018-06-21 23:23:29,617 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(943): Cleaning up temporary data for 04d71925444c7c823938ba3d9ad0504d 2018-06-21 23:23:29,619 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(954): Cleaning up detritus for 04d71925444c7c823938ba3d9ad0504d 2018-06-21 23:23:29,621 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(976): writing seq id for 04d71925444c7c823938ba3d9ad0504d 2018-06-21 23:23:29,622 INFO [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(980): Opened 04d71925444c7c823938ba3d9ad0504d; next sequenceid=11 2018-06-21 23:23:29,623 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(987): Running coprocessor post-open hooks for 04d71925444c7c823938ba3d9ad0504d 2018-06-21 23:23:29,625 INFO [PostOpenDeployTasks:04d71925444c7c823938ba3d9ad0504d] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,iii,1529623399386.04d71925444c7c823938ba3d9ad0504d. 2018-06-21 23:23:29,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=11, pid=48, ppid=44, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=04d71925444c7c823938ba3d9ad0504d, target=asf911.gq1.ygridcore.net,52507,1529623390228; rit=OPENING, location=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:29,627 DEBUG [PEWorker-5] assignment.RegionTransitionProcedure(354): Finishing pid=48, ppid=44, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=04d71925444c7c823938ba3d9ad0504d, target=asf911.gq1.ygridcore.net,52507,1529623390228; rit=OPENING, location=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:29,627 DEBUG [PostOpenDeployTasks:04d71925444c7c823938ba3d9ad0504d] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,iii,1529623399386.04d71925444c7c823938ba3d9ad0504d. 2018-06-21 23:23:29,627 INFO [PEWorker-5] assignment.RegionStateStore(198): pid=48 updating hbase:meta row=04d71925444c7c823938ba3d9ad0504d, regionState=OPEN, openSeqNum=11, regionLocation=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:29,627 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] handler.OpenRegionHandler(128): Opened multi_test_table,iii,1529623399386.04d71925444c7c823938ba3d9ad0504d. on asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:29,640 INFO [PEWorker-14] procedure2.ProcedureExecutor(1266): Finished pid=48, ppid=44, state=SUCCESS; AssignProcedure table=multi_test_table, region=04d71925444c7c823938ba3d9ad0504d, target=asf911.gq1.ygridcore.net,52507,1529623390228 in 523msec 2018-06-21 23:23:29,700 INFO [Time-limited test] master.HMaster(1539): balance hri=77a32d153e5d8148ca9d599c6a3cb695, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:29,781 INFO [PEWorker-3] procedure2.ProcedureExecutor(1635): Finished subprocedure(s) of pid=42, state=RUNNABLE; MoveRegionProcedure hri=fee67ae5d4f2b2be35fa516877a8d173, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,40767,1529623390148; resume parent processing. 2018-06-21 23:23:29,781 DEBUG [Time-limited test] procedure2.ProcedureExecutor(887): Stored pid=49, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=77a32d153e5d8148ca9d599c6a3cb695, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:29,781 INFO [PEWorker-15] procedure2.ProcedureExecutor(1635): Finished subprocedure(s) of pid=40, state=RUNNABLE; MoveRegionProcedure hri=8e2537011449fca6295a357cbde76e6c, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,58235,1529623390382; resume parent processing. 2018-06-21 23:23:29,782 INFO [Time-limited test] master.HMaster(1539): balance hri=b1a8d69ffdd43e80f30a919effdc7c4e, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:29,781 INFO [PEWorker-3] procedure2.ProcedureExecutor(1266): Finished pid=46, ppid=42, state=SUCCESS; AssignProcedure table=multi_test_table, region=fee67ae5d4f2b2be35fa516877a8d173, target=asf911.gq1.ygridcore.net,40767,1529623390148 in 534msec 2018-06-21 23:23:29,784 INFO [PEWorker-15] procedure2.ProcedureExecutor(1266): Finished pid=47, ppid=40, state=SUCCESS; AssignProcedure table=multi_test_table, region=8e2537011449fca6295a357cbde76e6c, target=asf911.gq1.ygridcore.net,58235,1529623390382 in 539msec 2018-06-21 23:23:29,784 INFO [PEWorker-11] procedure.MasterProcedureScheduler(696): pid=49, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=77a32d153e5d8148ca9d599c6a3cb695, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,56613,1529623390297 checking lock on 77a32d153e5d8148ca9d599c6a3cb695 2018-06-21 23:23:29,785 INFO [PEWorker-11] procedure2.ProcedureExecutor(1516): Initialized subprocedures=[{pid=50, ppid=49, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=multi_test_table, region=77a32d153e5d8148ca9d599c6a3cb695, server=asf911.gq1.ygridcore.net,55146,1529623388534}] 2018-06-21 23:23:29,863 INFO [PEWorker-5] procedure2.ProcedureExecutor(1635): Finished subprocedure(s) of pid=44, state=RUNNABLE; MoveRegionProcedure hri=04d71925444c7c823938ba3d9ad0504d, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,52507,1529623390228; resume parent processing. 2018-06-21 23:23:29,863 DEBUG [Time-limited test] procedure2.ProcedureExecutor(887): Stored pid=51, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=b1a8d69ffdd43e80f30a919effdc7c4e, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:29,863 INFO [PEWorker-11] procedure.MasterProcedureScheduler(696): pid=50, ppid=49, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=multi_test_table, region=77a32d153e5d8148ca9d599c6a3cb695, server=asf911.gq1.ygridcore.net,55146,1529623388534 checking lock on 77a32d153e5d8148ca9d599c6a3cb695 2018-06-21 23:23:29,863 INFO [PEWorker-5] procedure2.ProcedureExecutor(1266): Finished pid=48, ppid=44, state=SUCCESS; AssignProcedure table=multi_test_table, region=04d71925444c7c823938ba3d9ad0504d, target=asf911.gq1.ygridcore.net,52507,1529623390228 in 523msec 2018-06-21 23:23:29,863 INFO [PEWorker-16] procedure2.ProcedureExecutor(1266): Finished pid=42, state=SUCCESS; MoveRegionProcedure hri=fee67ae5d4f2b2be35fa516877a8d173, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,40767,1529623390148 in 1.6370sec 2018-06-21 23:23:29,866 INFO [PEWorker-5] procedure.MasterProcedureScheduler(696): pid=51, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=b1a8d69ffdd43e80f30a919effdc7c4e, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,58235,1529623390382 checking lock on b1a8d69ffdd43e80f30a919effdc7c4e 2018-06-21 23:23:29,863 INFO [PEWorker-10] procedure2.ProcedureExecutor(1266): Finished pid=40, state=SUCCESS; MoveRegionProcedure hri=8e2537011449fca6295a357cbde76e6c, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,58235,1529623390382 in 1.8130sec 2018-06-21 23:23:29,864 INFO [PEWorker-11] assignment.RegionStateStore(198): pid=50 updating hbase:meta row=77a32d153e5d8148ca9d599c6a3cb695, regionState=CLOSING, regionLocation=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:29,866 INFO [PEWorker-5] procedure2.ProcedureExecutor(1516): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=multi_test_table, region=b1a8d69ffdd43e80f30a919effdc7c4e, server=asf911.gq1.ygridcore.net,55146,1529623388534}] 2018-06-21 23:23:29,870 INFO [PEWorker-11] assignment.RegionTransitionProcedure(241): Dispatch pid=50, ppid=49, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=multi_test_table, region=77a32d153e5d8148ca9d599c6a3cb695, server=asf911.gq1.ygridcore.net,55146,1529623388534; rit=CLOSING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:30,021 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(119): Fallback to compat rpc execution for serverName=asf911.gq1.ygridcore.net,55146,1529623388534 version=0 2018-06-21 23:23:30,022 INFO [RSProcedureDispatcher-pool3-t25] regionserver.RSRpcServices(1604): Close 77a32d153e5d8148ca9d599c6a3cb695, moving to asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:30,024 DEBUG [RS_CLOSE_REGION-master/asf911:0-0] regionserver.HRegion(1520): Closing 77a32d153e5d8148ca9d599c6a3cb695, disabling compactions & flushes 2018-06-21 23:23:30,025 DEBUG [RS_CLOSE_REGION-master/asf911:0-0] regionserver.HRegion(1560): Updates disabled for region multi_test_table,xxx,1529623399386.77a32d153e5d8148ca9d599c6a3cb695. 2018-06-21 23:23:30,025 INFO [RS_CLOSE_REGION-master/asf911:0-0] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=1.86 KB heapSize=4.13 KB 2018-06-21 23:23:30,061 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741880_1056{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:30,061 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741880_1056{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:30,061 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741880_1056{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:30,062 INFO [RS_CLOSE_REGION-master/asf911:0-0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=1.86 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/77a32d153e5d8148ca9d599c6a3cb695/.tmp/family/5c80e52cd742440585096852105fc669 2018-06-21 23:23:30,086 INFO [RS_CLOSE_REGION-master/asf911:0-0] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5c80e52cd742440585096852105fc669 2018-06-21 23:23:30,088 DEBUG [RS_CLOSE_REGION-master/asf911:0-0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/77a32d153e5d8148ca9d599c6a3cb695/.tmp/family/5c80e52cd742440585096852105fc669 as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/77a32d153e5d8148ca9d599c6a3cb695/family/5c80e52cd742440585096852105fc669 2018-06-21 23:23:30,098 INFO [RS_CLOSE_REGION-master/asf911:0-0] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5c80e52cd742440585096852105fc669 2018-06-21 23:23:30,099 INFO [RS_CLOSE_REGION-master/asf911:0-0] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/77a32d153e5d8148ca9d599c6a3cb695/family/5c80e52cd742440585096852105fc669, entries=15, sequenceid=11, filesize=5.4 K 2018-06-21 23:23:30,102 INFO [RS_CLOSE_REGION-master/asf911:0-0] regionserver.HRegion(2769): Finished flush of dataSize ~1.86 KB/1907, heapSize ~4.34 KB/4440, currentSize=0 B/0 for 77a32d153e5d8148ca9d599c6a3cb695 in 77ms, sequenceid=11, compaction requested=false 2018-06-21 23:23:30,111 DEBUG [RS_CLOSE_REGION-master/asf911:0-0] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/77a32d153e5d8148ca9d599c6a3cb695/recovered.edits/14.seqid, newMaxSeqId=14, maxSeqId=1 2018-06-21 23:23:30,116 INFO [RS_CLOSE_REGION-master/asf911:0-0] regionserver.HRegion(1674): Closed multi_test_table,xxx,1529623399386.77a32d153e5d8148ca9d599c6a3cb695. 2018-06-21 23:23:30,116 INFO [RS_CLOSE_REGION-master/asf911:0-0] regionserver.HRegionServer(3426): Adding 77a32d153e5d8148ca9d599c6a3cb695 move to asf911.gq1.ygridcore.net,56613,1529623390297 record at close sequenceid=11 2018-06-21 23:23:30,117 DEBUG [RS_CLOSE_REGION-master/asf911:0-0] assignment.RegionTransitionProcedure(264): Received report CLOSED seqId=-1, pid=50, ppid=49, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=multi_test_table, region=77a32d153e5d8148ca9d599c6a3cb695, server=asf911.gq1.ygridcore.net,55146,1529623388534; rit=CLOSING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:30,117 DEBUG [RS_CLOSE_REGION-master/asf911:0-0] handler.CloseRegionHandler(124): Closed multi_test_table,xxx,1529623399386.77a32d153e5d8148ca9d599c6a3cb695. 2018-06-21 23:23:30,119 DEBUG [PEWorker-4] assignment.RegionTransitionProcedure(354): Finishing pid=50, ppid=49, state=RUNNABLE:REGION_TRANSITION_FINISH; UnassignProcedure table=multi_test_table, region=77a32d153e5d8148ca9d599c6a3cb695, server=asf911.gq1.ygridcore.net,55146,1529623388534; rit=CLOSING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:30,119 INFO [PEWorker-4] assignment.RegionStateStore(198): pid=50 updating hbase:meta row=77a32d153e5d8148ca9d599c6a3cb695, regionState=CLOSED 2018-06-21 23:23:30,119 INFO [PEWorker-1] procedure2.ProcedureExecutor(1266): Finished pid=44, state=SUCCESS; MoveRegionProcedure hri=04d71925444c7c823938ba3d9ad0504d, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,52507,1529623390228 in 1.6200sec 2018-06-21 23:23:30,121 INFO [PEWorker-11] procedure.MasterProcedureScheduler(696): pid=52, ppid=51, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=multi_test_table, region=b1a8d69ffdd43e80f30a919effdc7c4e, server=asf911.gq1.ygridcore.net,55146,1529623388534 checking lock on b1a8d69ffdd43e80f30a919effdc7c4e 2018-06-21 23:23:30,121 INFO [PEWorker-11] assignment.RegionStateStore(198): pid=52 updating hbase:meta row=b1a8d69ffdd43e80f30a919effdc7c4e, regionState=CLOSING, regionLocation=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:30,126 INFO [PEWorker-11] assignment.RegionTransitionProcedure(241): Dispatch pid=52, ppid=51, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=multi_test_table, region=b1a8d69ffdd43e80f30a919effdc7c4e, server=asf911.gq1.ygridcore.net,55146,1529623388534; rit=CLOSING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:30,276 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(119): Fallback to compat rpc execution for serverName=asf911.gq1.ygridcore.net,55146,1529623388534 version=0 2018-06-21 23:23:30,277 INFO [RSProcedureDispatcher-pool3-t27] regionserver.RSRpcServices(1604): Close b1a8d69ffdd43e80f30a919effdc7c4e, moving to asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:30,280 DEBUG [RS_CLOSE_REGION-master/asf911:0-1] regionserver.HRegion(1520): Closing b1a8d69ffdd43e80f30a919effdc7c4e, disabling compactions & flushes 2018-06-21 23:23:30,280 DEBUG [RS_CLOSE_REGION-master/asf911:0-1] regionserver.HRegion(1560): Updates disabled for region multi_test_table,nnn,1529623399386.b1a8d69ffdd43e80f30a919effdc7c4e. 2018-06-21 23:23:30,280 INFO [RS_CLOSE_REGION-master/asf911:0-1] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=1.60 KB heapSize=3.28 KB 2018-06-21 23:23:30,306 INFO [PEWorker-4] procedure2.ProcedureExecutor(1635): Finished subprocedure(s) of pid=49, state=RUNNABLE:MOVE_REGION_ASSIGN; MoveRegionProcedure hri=77a32d153e5d8148ca9d599c6a3cb695, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,56613,1529623390297; resume parent processing. 2018-06-21 23:23:30,306 INFO [PEWorker-6] procedure2.ProcedureExecutor(1516): Initialized subprocedures=[{pid=53, ppid=49, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=77a32d153e5d8148ca9d599c6a3cb695, target=asf911.gq1.ygridcore.net,56613,1529623390297}] 2018-06-21 23:23:30,306 INFO [PEWorker-4] procedure2.ProcedureExecutor(1266): Finished pid=50, ppid=49, state=SUCCESS; UnassignProcedure table=multi_test_table, region=77a32d153e5d8148ca9d599c6a3cb695, server=asf911.gq1.ygridcore.net,55146,1529623388534 in 341msec 2018-06-21 23:23:30,324 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741881_1057{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:30,324 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741881_1057{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|FINALIZED]]} size 0 2018-06-21 23:23:30,325 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741881_1057 size 5232 2018-06-21 23:23:30,326 INFO [RS_CLOSE_REGION-master/asf911:0-1] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=1.60 KB at sequenceid=7 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/b1a8d69ffdd43e80f30a919effdc7c4e/.tmp/family/845531426a164ec38537a9496191e128 2018-06-21 23:23:30,339 DEBUG [RS_CLOSE_REGION-master/asf911:0-1] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/b1a8d69ffdd43e80f30a919effdc7c4e/.tmp/family/845531426a164ec38537a9496191e128 as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/b1a8d69ffdd43e80f30a919effdc7c4e/family/845531426a164ec38537a9496191e128 2018-06-21 23:23:30,373 INFO [RS_CLOSE_REGION-master/asf911:0-1] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/b1a8d69ffdd43e80f30a919effdc7c4e/family/845531426a164ec38537a9496191e128, entries=10, sequenceid=7, filesize=5.1 K 2018-06-21 23:23:30,380 INFO [RS_CLOSE_REGION-master/asf911:0-1] regionserver.HRegion(2769): Finished flush of dataSize ~1.60 KB/1638, heapSize ~3.49 KB/3576, currentSize=0 B/0 for b1a8d69ffdd43e80f30a919effdc7c4e in 100ms, sequenceid=7, compaction requested=false 2018-06-21 23:23:30,390 DEBUG [RS_CLOSE_REGION-master/asf911:0-1] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/b1a8d69ffdd43e80f30a919effdc7c4e/recovered.edits/10.seqid, newMaxSeqId=10, maxSeqId=1 2018-06-21 23:23:30,393 INFO [RS_CLOSE_REGION-master/asf911:0-1] regionserver.HRegion(1674): Closed multi_test_table,nnn,1529623399386.b1a8d69ffdd43e80f30a919effdc7c4e. 2018-06-21 23:23:30,393 INFO [RS_CLOSE_REGION-master/asf911:0-1] regionserver.HRegionServer(3426): Adding b1a8d69ffdd43e80f30a919effdc7c4e move to asf911.gq1.ygridcore.net,58235,1529623390382 record at close sequenceid=7 2018-06-21 23:23:30,394 DEBUG [RS_CLOSE_REGION-master/asf911:0-1] assignment.RegionTransitionProcedure(264): Received report CLOSED seqId=-1, pid=52, ppid=51, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=multi_test_table, region=b1a8d69ffdd43e80f30a919effdc7c4e, server=asf911.gq1.ygridcore.net,55146,1529623388534; rit=CLOSING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:30,394 DEBUG [RS_CLOSE_REGION-master/asf911:0-1] handler.CloseRegionHandler(124): Closed multi_test_table,nnn,1529623399386.b1a8d69ffdd43e80f30a919effdc7c4e. 2018-06-21 23:23:30,394 DEBUG [PEWorker-12] assignment.RegionTransitionProcedure(354): Finishing pid=52, ppid=51, state=RUNNABLE:REGION_TRANSITION_FINISH; UnassignProcedure table=multi_test_table, region=b1a8d69ffdd43e80f30a919effdc7c4e, server=asf911.gq1.ygridcore.net,55146,1529623388534; rit=CLOSING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:30,395 INFO [PEWorker-12] assignment.RegionStateStore(198): pid=52 updating hbase:meta row=b1a8d69ffdd43e80f30a919effdc7c4e, regionState=CLOSED 2018-06-21 23:23:30,404 INFO [PEWorker-6] procedure.MasterProcedureScheduler(696): pid=53, ppid=49, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=77a32d153e5d8148ca9d599c6a3cb695, target=asf911.gq1.ygridcore.net,56613,1529623390297 checking lock on 77a32d153e5d8148ca9d599c6a3cb695 2018-06-21 23:23:30,408 INFO [PEWorker-6] assignment.AssignProcedure(218): Starting pid=53, ppid=49, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=77a32d153e5d8148ca9d599c6a3cb695, target=asf911.gq1.ygridcore.net,56613,1529623390297; rit=OFFLINE, location=asf911.gq1.ygridcore.net,56613,1529623390297; forceNewPlan=false, retain=false 2018-06-21 23:23:30,559 INFO [master/asf911:0] balancer.BaseLoadBalancer(1497): Reassigned 1 regions. 1 retained the pre-restart assignment. 2018-06-21 23:23:30,560 INFO [PEWorker-2] assignment.RegionStateStore(198): pid=53 updating hbase:meta row=77a32d153e5d8148ca9d599c6a3cb695, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:30,562 INFO [PEWorker-12] procedure2.ProcedureExecutor(1635): Finished subprocedure(s) of pid=51, state=RUNNABLE:MOVE_REGION_ASSIGN; MoveRegionProcedure hri=b1a8d69ffdd43e80f30a919effdc7c4e, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,58235,1529623390382; resume parent processing. 2018-06-21 23:23:30,562 INFO [PEWorker-14] procedure2.ProcedureExecutor(1516): Initialized subprocedures=[{pid=54, ppid=51, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=b1a8d69ffdd43e80f30a919effdc7c4e, target=asf911.gq1.ygridcore.net,58235,1529623390382}] 2018-06-21 23:23:30,562 INFO [PEWorker-12] procedure2.ProcedureExecutor(1266): Finished pid=52, ppid=51, state=SUCCESS; UnassignProcedure table=multi_test_table, region=b1a8d69ffdd43e80f30a919effdc7c4e, server=asf911.gq1.ygridcore.net,55146,1529623388534 in 533msec 2018-06-21 23:23:30,564 INFO [PEWorker-2] assignment.RegionTransitionProcedure(241): Dispatch pid=53, ppid=49, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=77a32d153e5d8148ca9d599c6a3cb695, target=asf911.gq1.ygridcore.net,56613,1529623390297; rit=OPENING, location=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:30,671 INFO [PEWorker-14] procedure.MasterProcedureScheduler(696): pid=54, ppid=51, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=b1a8d69ffdd43e80f30a919effdc7c4e, target=asf911.gq1.ygridcore.net,58235,1529623390382 checking lock on b1a8d69ffdd43e80f30a919effdc7c4e 2018-06-21 23:23:30,673 INFO [PEWorker-14] assignment.AssignProcedure(218): Starting pid=54, ppid=51, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=b1a8d69ffdd43e80f30a919effdc7c4e, target=asf911.gq1.ygridcore.net,58235,1529623390382; rit=OFFLINE, location=asf911.gq1.ygridcore.net,58235,1529623390382; forceNewPlan=false, retain=false 2018-06-21 23:23:30,720 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=56613] regionserver.RSRpcServices(1980): Open multi_test_table,xxx,1529623399386.77a32d153e5d8148ca9d599c6a3cb695. 2018-06-21 23:23:30,726 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7094): Opening region: {ENCODED => 77a32d153e5d8148ca9d599c6a3cb695, NAME => 'multi_test_table,xxx,1529623399386.77a32d153e5d8148ca9d599c6a3cb695.', STARTKEY => 'xxx', ENDKEY => 'yyy'} 2018-06-21 23:23:30,726 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table 77a32d153e5d8148ca9d599c6a3cb695 2018-06-21 23:23:30,727 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(827): Instantiated multi_test_table,xxx,1529623399386.77a32d153e5d8148ca9d599c6a3cb695.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:30,727 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7134): checking encryption for 77a32d153e5d8148ca9d599c6a3cb695 2018-06-21 23:23:30,727 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7139): checking classloading for 77a32d153e5d8148ca9d599c6a3cb695 2018-06-21 23:23:30,731 DEBUG [StoreOpener-77a32d153e5d8148ca9d599c6a3cb695-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/77a32d153e5d8148ca9d599c6a3cb695/family 2018-06-21 23:23:30,731 DEBUG [StoreOpener-77a32d153e5d8148ca9d599c6a3cb695-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/77a32d153e5d8148ca9d599c6a3cb695/family 2018-06-21 23:23:30,732 INFO [StoreOpener-77a32d153e5d8148ca9d599c6a3cb695-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=5, currentSize=750.81 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=750.81 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:30,733 INFO [StoreOpener-77a32d153e5d8148ca9d599c6a3cb695-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:30,751 INFO [StoreFileOpenerThread-family-1] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5c80e52cd742440585096852105fc669 2018-06-21 23:23:30,751 DEBUG [StoreOpener-77a32d153e5d8148ca9d599c6a3cb695-1] regionserver.HStore(581): loaded hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/77a32d153e5d8148ca9d599c6a3cb695/family/5c80e52cd742440585096852105fc669 2018-06-21 23:23:30,751 INFO [StoreOpener-77a32d153e5d8148ca9d599c6a3cb695-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:30,751 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(923): replaying wal for 77a32d153e5d8148ca9d599c6a3cb695 2018-06-21 23:23:30,754 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/77a32d153e5d8148ca9d599c6a3cb695 2018-06-21 23:23:30,754 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(931): stopping wal replay for 77a32d153e5d8148ca9d599c6a3cb695 2018-06-21 23:23:30,754 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(943): Cleaning up temporary data for 77a32d153e5d8148ca9d599c6a3cb695 2018-06-21 23:23:30,763 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(954): Cleaning up detritus for 77a32d153e5d8148ca9d599c6a3cb695 2018-06-21 23:23:30,766 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(976): writing seq id for 77a32d153e5d8148ca9d599c6a3cb695 2018-06-21 23:23:30,767 INFO [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(980): Opened 77a32d153e5d8148ca9d599c6a3cb695; next sequenceid=15 2018-06-21 23:23:30,767 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(987): Running coprocessor post-open hooks for 77a32d153e5d8148ca9d599c6a3cb695 2018-06-21 23:23:30,771 INFO [PostOpenDeployTasks:77a32d153e5d8148ca9d599c6a3cb695] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,xxx,1529623399386.77a32d153e5d8148ca9d599c6a3cb695. 2018-06-21 23:23:30,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=15, pid=53, ppid=49, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=77a32d153e5d8148ca9d599c6a3cb695, target=asf911.gq1.ygridcore.net,56613,1529623390297; rit=OPENING, location=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:30,773 DEBUG [PEWorker-15] assignment.RegionTransitionProcedure(354): Finishing pid=53, ppid=49, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=77a32d153e5d8148ca9d599c6a3cb695, target=asf911.gq1.ygridcore.net,56613,1529623390297; rit=OPENING, location=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:30,773 INFO [PEWorker-15] assignment.RegionStateStore(198): pid=53 updating hbase:meta row=77a32d153e5d8148ca9d599c6a3cb695, regionState=OPEN, openSeqNum=15, regionLocation=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:30,774 DEBUG [PostOpenDeployTasks:77a32d153e5d8148ca9d599c6a3cb695] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,xxx,1529623399386.77a32d153e5d8148ca9d599c6a3cb695. 2018-06-21 23:23:30,774 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] handler.OpenRegionHandler(128): Opened multi_test_table,xxx,1529623399386.77a32d153e5d8148ca9d599c6a3cb695. on asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:30,823 INFO [master/asf911:0] balancer.BaseLoadBalancer(1497): Reassigned 1 regions. 1 retained the pre-restart assignment. 2018-06-21 23:23:30,824 INFO [PEWorker-8] assignment.RegionStateStore(198): pid=54 updating hbase:meta row=b1a8d69ffdd43e80f30a919effdc7c4e, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:30,828 INFO [PEWorker-8] assignment.RegionTransitionProcedure(241): Dispatch pid=54, ppid=51, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=b1a8d69ffdd43e80f30a919effdc7c4e, target=asf911.gq1.ygridcore.net,58235,1529623390382; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:30,873 INFO [PEWorker-15] procedure2.ProcedureExecutor(1635): Finished subprocedure(s) of pid=49, state=RUNNABLE; MoveRegionProcedure hri=77a32d153e5d8148ca9d599c6a3cb695, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,56613,1529623390297; resume parent processing. 2018-06-21 23:23:30,874 INFO [PEWorker-15] procedure2.ProcedureExecutor(1266): Finished pid=53, ppid=49, state=SUCCESS; AssignProcedure table=multi_test_table, region=77a32d153e5d8148ca9d599c6a3cb695, target=asf911.gq1.ygridcore.net,56613,1529623390297 in 470msec 2018-06-21 23:23:30,949 INFO [PEWorker-7] procedure2.ProcedureExecutor(1266): Finished pid=49, state=SUCCESS; MoveRegionProcedure hri=77a32d153e5d8148ca9d599c6a3cb695, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,56613,1529623390297 in 1.1710sec 2018-06-21 23:23:30,980 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58235] regionserver.RSRpcServices(1980): Open multi_test_table,nnn,1529623399386.b1a8d69ffdd43e80f30a919effdc7c4e. 2018-06-21 23:23:30,988 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7094): Opening region: {ENCODED => b1a8d69ffdd43e80f30a919effdc7c4e, NAME => 'multi_test_table,nnn,1529623399386.b1a8d69ffdd43e80f30a919effdc7c4e.', STARTKEY => 'nnn', ENDKEY => 'ooo'} 2018-06-21 23:23:30,988 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table b1a8d69ffdd43e80f30a919effdc7c4e 2018-06-21 23:23:30,988 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(827): Instantiated multi_test_table,nnn,1529623399386.b1a8d69ffdd43e80f30a919effdc7c4e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:30,989 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7134): checking encryption for b1a8d69ffdd43e80f30a919effdc7c4e 2018-06-21 23:23:30,989 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7139): checking classloading for b1a8d69ffdd43e80f30a919effdc7c4e 2018-06-21 23:23:30,993 DEBUG [StoreOpener-b1a8d69ffdd43e80f30a919effdc7c4e-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/b1a8d69ffdd43e80f30a919effdc7c4e/family 2018-06-21 23:23:30,993 DEBUG [StoreOpener-b1a8d69ffdd43e80f30a919effdc7c4e-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/b1a8d69ffdd43e80f30a919effdc7c4e/family 2018-06-21 23:23:30,994 INFO [StoreOpener-b1a8d69ffdd43e80f30a919effdc7c4e-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=5, currentSize=750.81 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=750.81 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:30,994 INFO [StoreOpener-b1a8d69ffdd43e80f30a919effdc7c4e-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:31,008 DEBUG [StoreOpener-b1a8d69ffdd43e80f30a919effdc7c4e-1] regionserver.HStore(581): loaded hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/b1a8d69ffdd43e80f30a919effdc7c4e/family/845531426a164ec38537a9496191e128 2018-06-21 23:23:31,009 INFO [StoreOpener-b1a8d69ffdd43e80f30a919effdc7c4e-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:31,009 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(923): replaying wal for b1a8d69ffdd43e80f30a919effdc7c4e 2018-06-21 23:23:31,011 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/b1a8d69ffdd43e80f30a919effdc7c4e 2018-06-21 23:23:31,011 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(931): stopping wal replay for b1a8d69ffdd43e80f30a919effdc7c4e 2018-06-21 23:23:31,011 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(943): Cleaning up temporary data for b1a8d69ffdd43e80f30a919effdc7c4e 2018-06-21 23:23:31,012 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(954): Cleaning up detritus for b1a8d69ffdd43e80f30a919effdc7c4e 2018-06-21 23:23:31,014 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(976): writing seq id for b1a8d69ffdd43e80f30a919effdc7c4e 2018-06-21 23:23:31,015 INFO [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(980): Opened b1a8d69ffdd43e80f30a919effdc7c4e; next sequenceid=11 2018-06-21 23:23:31,015 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(987): Running coprocessor post-open hooks for b1a8d69ffdd43e80f30a919effdc7c4e 2018-06-21 23:23:31,018 INFO [PostOpenDeployTasks:b1a8d69ffdd43e80f30a919effdc7c4e] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,nnn,1529623399386.b1a8d69ffdd43e80f30a919effdc7c4e. 2018-06-21 23:23:31,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=11, pid=54, ppid=51, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=b1a8d69ffdd43e80f30a919effdc7c4e, target=asf911.gq1.ygridcore.net,58235,1529623390382; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:31,019 DEBUG [PEWorker-16] assignment.RegionTransitionProcedure(354): Finishing pid=54, ppid=51, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=b1a8d69ffdd43e80f30a919effdc7c4e, target=asf911.gq1.ygridcore.net,58235,1529623390382; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:31,020 DEBUG [PostOpenDeployTasks:b1a8d69ffdd43e80f30a919effdc7c4e] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,nnn,1529623399386.b1a8d69ffdd43e80f30a919effdc7c4e. 2018-06-21 23:23:31,020 INFO [PEWorker-16] assignment.RegionStateStore(198): pid=54 updating hbase:meta row=b1a8d69ffdd43e80f30a919effdc7c4e, regionState=OPEN, openSeqNum=11, regionLocation=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:31,020 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] handler.OpenRegionHandler(128): Opened multi_test_table,nnn,1529623399386.b1a8d69ffdd43e80f30a919effdc7c4e. on asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:31,067 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [3,000] milli-secs(wait.for.ratio=[1]) 2018-06-21 23:23:31,204 INFO [PEWorker-16] procedure2.ProcedureExecutor(1635): Finished subprocedure(s) of pid=51, state=RUNNABLE; MoveRegionProcedure hri=b1a8d69ffdd43e80f30a919effdc7c4e, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,58235,1529623390382; resume parent processing. 2018-06-21 23:23:31,205 INFO [PEWorker-16] procedure2.ProcedureExecutor(1266): Finished pid=54, ppid=51, state=SUCCESS; AssignProcedure table=multi_test_table, region=b1a8d69ffdd43e80f30a919effdc7c4e, target=asf911.gq1.ygridcore.net,58235,1529623390382 in 460msec 2018-06-21 23:23:31,273 INFO [PEWorker-10] procedure2.ProcedureExecutor(1266): Finished pid=51, state=SUCCESS; MoveRegionProcedure hri=b1a8d69ffdd43e80f30a919effdc7c4e, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,58235,1529623390382 in 1.4180sec 2018-06-21 23:23:31,965 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@69ed135] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:53553 to delete [blk_1073741866_1042, blk_1073741867_1043, blk_1073741869_1045] 2018-06-21 23:23:31,965 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@69ed135] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:54793 to delete [blk_1073741867_1043, blk_1073741868_1044, blk_1073741870_1046] 2018-06-21 23:23:32,119 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-06-21 23:23:34,066 WARN [Time-limited test] hbase.Waiter(213): Waiting timed out after [3,000] msec 2018-06-21 23:23:34,066 DEBUG [Time-limited test] hbase.HBaseTestingUtility(3284): Waiting until all regions of table multi_test_table get assigned. Timeout = 60000ms 2018-06-21 23:23:34,067 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2018-06-21 23:23:34,073 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=2,queue=0,port=40767] regionserver.StoreScanner(1000): Switch to stream read (scanned=32905 bytes) of info 2018-06-21 23:23:34,096 INFO [Time-limited test] hbase.HBaseTestingUtility(3336): All regions for table multi_test_table assigned to meta. Checking AM states. 2018-06-21 23:23:34,097 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2018-06-21 23:23:34,097 INFO [Time-limited test] hbase.HBaseTestingUtility(3356): All regions for table multi_test_table assigned. 2018-06-21 23:23:34,098 INFO [Time-limited test] client.TestMultiParallel(133): before done 2018-06-21 23:23:34,098 INFO [Time-limited test] client.TestMultiParallel(288): test=testFlushCommitsWithAbort 2018-06-21 23:23:34,099 INFO [Time-limited test] client.TestMultiParallel(299): get new table 2018-06-21 23:23:34,099 INFO [Time-limited test] client.TestMultiParallel(302): constructPutRequests 2018-06-21 23:23:34,109 DEBUG [Default-IPC-NioEventLoopGroup-9-15] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:59970 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:59970 2018-06-21 23:23:34,282 INFO [Time-limited test] client.TestMultiParallel(305): puts 2018-06-21 23:23:34,282 INFO [Time-limited test] hbase.LocalHBaseCluster(270): Not alive RS:0;asf911:59970 2018-06-21 23:23:34,282 INFO [Time-limited test] hbase.LocalHBaseCluster(270): Not alive RS:0;asf911:59970 2018-06-21 23:23:34,282 ERROR [Time-limited test] helpers.MarkerIgnoringBase(159): ***** ABORTING region server asf911.gq1.ygridcore.net,40767,1529623390148: Aborting for tests ***** java.lang.Exception: doTestFlushCommits at org.apache.hadoop.hbase.client.TestMultiParallel.doTestFlushCommits(TestMultiParallel.java:312) at org.apache.hadoop.hbase.client.TestMultiParallel.testFlushCommitsWithAbort(TestMultiParallel.java:289) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) 2018-06-21 23:23:34,283 ERROR [Time-limited test] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.client.TestMultiParallel$MyMasterObserver, org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-06-21 23:23:34,295 INFO [Time-limited test] regionserver.HRegionServer(2371): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "Verbose" : false, "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 1312292864, "init" : 792723456, "max" : 2609905664, "used" : 544646064 }, "NonHeapMemoryUsage" : { "committed" : 135643136, "init" : 2555904, "max" : -1, "used" : 133096120 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "queueSize" : 0, "numCallsInGeneralQueue" : 0, "numCallsInReplicationQueue" : 0, "numCallsInPriorityQueue" : 0, "numOpenConnections" : -1, "numActiveHandler" : 0, "numActiveGeneralHandler" : 0, "numActivePriorityHandler" : 0, "numActiveReplicationHandler" : 0, "numGeneralCallsDropped" : 0, "numLifoModeSwitches" : 0, "numCallsInWriteQueue" : 0, "numCallsInReadQueue" : 0, "numCallsInScanQueue" : 0, "numActiveWriteHandler" : 0, "numActiveReadHandler" : 0, "numActiveScanHandler" : 0, "receivedBytes" : 217845, "exceptions.RegionMovedException" : 0, "exceptions.multiResponseTooLarge" : 0, "authenticationSuccesses" : 0, "authorizationFailures" : 0, "TotalCallTime_num_ops" : 651, "TotalCallTime_min" : 0, "TotalCallTime_max" : 59, "TotalCallTime_mean" : 5, "TotalCallTime_25th_percentile" : 1, "TotalCallTime_median" : 2, "TotalCallTime_75th_percentile" : 3, "TotalCallTime_90th_percentile" : 18, "TotalCallTime_95th_percentile" : 43, "TotalCallTime_98th_percentile" : 56, "TotalCallTime_99th_percentile" : 59, "TotalCallTime_99.9th_percentile" : 59, "TotalCallTime_TimeRangeCount_0-1" : 23, "TotalCallTime_TimeRangeCount_1-3" : 21, "TotalCallTime_TimeRangeCount_3-10" : 6, "TotalCallTime_TimeRangeCount_10-30" : 3, "TotalCallTime_TimeRangeCount_30-100" : 4, "TotalCallTime_TimeRangeCount_600000-inf" : 594, "exceptions.RegionTooBusyException" : 0, "exceptions.FailedSanityCheckException" : 0, "ResponseSize_num_ops" : 651, "ResponseSize_min" : 0, "ResponseSize_max" : 4634, "ResponseSize_mean" : 640, "ResponseSize_25th_percentile" : 12, "ResponseSize_median" : 25, "ResponseSize_75th_percentile" : 78, "ResponseSize_90th_percentile" : 3926, "ResponseSize_95th_percentile" : 3942, "ResponseSize_98th_percentile" : 3952, "ResponseSize_99th_percentile" : 4626, "ResponseSize_99.9th_percentile" : 4633, "ResponseSize_SizeRangeCount_0-10" : 30, "ResponseSize_SizeRangeCount_10-100" : 17, "ResponseSize_SizeRangeCount_100-1000" : 1, "ResponseSize_SizeRangeCount_1000-10000" : 9, "ResponseSize_SizeRangeCount_100000000-inf" : 594, "exceptions.UnknownScannerException" : 0, "exceptions.OutOfOrderScannerNextException" : 0, "exceptions" : 2, "ProcessCallTime_num_ops" : 651, "ProcessCallTime_min" : 0, "ProcessCallTime_max" : 59, "ProcessCallTime_mean" : 5, "ProcessCallTime_25th_percentile" : 1, "ProcessCallTime_median" : 2, "ProcessCallTime_75th_percentile" : 3, "ProcessCallTime_90th_percentile" : 18, "ProcessCallTime_95th_percentile" : 43, "ProcessCallTime_98th_percentile" : 56, "ProcessCallTime_99th_percentile" : 59, "ProcessCallTime_99.9th_percentile" : 59, "ProcessCallTime_TimeRangeCount_0-1" : 25, "ProcessCallTime_TimeRangeCount_1-3" : 20, "ProcessCallTime_TimeRangeCount_3-10" : 5, "ProcessCallTime_TimeRangeCount_10-30" : 3, "ProcessCallTime_TimeRangeCount_30-100" : 4, "ProcessCallTime_TimeRangeCount_600000-inf" : 594, "authenticationFallbacks" : 0, "exceptions.NotServingRegionException" : 0, "exceptions.callQueueTooBig" : 0, "authorizationSuccesses" : 17, "exceptions.ScannerResetException" : 0, "RequestSize_num_ops" : 651, "RequestSize_min" : 40, "RequestSize_max" : 4498, "RequestSize_mean" : 485, "RequestSize_25th_percentile" : 100, "RequestSize_median" : 136, "RequestSize_75th_percentile" : 312, "RequestSize_90th_percentile" : 1120, "RequestSize_95th_percentile" : 3745, "RequestSize_98th_percentile" : 3769, "RequestSize_99th_percentile" : 4497, "RequestSize_99.9th_percentile" : 4497, "RequestSize_SizeRangeCount_10-100" : 25, "RequestSize_SizeRangeCount_100-1000" : 25, "RequestSize_SizeRangeCount_1000-10000" : 7, "RequestSize_SizeRangeCount_100000000-inf" : 594, "sentBytes" : 377154, "QueueCallTime_num_ops" : 651, "QueueCallTime_min" : 0, "QueueCallTime_max" : 2, "QueueCallTime_mean" : 0, "QueueCallTime_25th_percentile" : 0, "QueueCallTime_median" : 0, "QueueCallTime_75th_percentile" : 0, "QueueCallTime_90th_percentile" : 0, "QueueCallTime_95th_percentile" : 1, "QueueCallTime_98th_percentile" : 1, "QueueCallTime_99th_percentile" : 2, "QueueCallTime_99.9th_percentile" : 2, "QueueCallTime_TimeRangeCount_0-1" : 56, "QueueCallTime_TimeRangeCount_1-3" : 1, "QueueCallTime_TimeRangeCount_600000-inf" : 594, "authenticationFailures" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "sink.appliedHFiles" : 0, "sink.appliedOps" : 0, "Sink.ageOfLastAppliedOp_num_ops" : 0, "Sink.ageOfLastAppliedOp_min" : 0, "Sink.ageOfLastAppliedOp_max" : 0, "Sink.ageOfLastAppliedOp_mean" : 0, "Sink.ageOfLastAppliedOp_25th_percentile" : 0, "Sink.ageOfLastAppliedOp_median" : 0, "Sink.ageOfLastAppliedOp_75th_percentile" : 0, "Sink.ageOfLastAppliedOp_90th_percentile" : 0, "Sink.ageOfLastAppliedOp_95th_percentile" : 0, "Sink.ageOfLastAppliedOp_98th_percentile" : 0, "Sink.ageOfLastAppliedOp_99th_percentile" : 0, "Sink.ageOfLastAppliedOp_99.9th_percentile" : 0, "sink.appliedBatches" : 0 } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.zookeeperQuorum" : "localhost:58989", "tag.serverName" : "asf911.gq1.ygridcore.net,58235,1529623390382", "tag.clusterId" : "2ed7179d-84c9-432a-b6b5-556343aa7690", "tag.Context" : "regionserver", "tag.Hostname" : "asf911.gq1.ygridcore.net", "regionCount" : 9, "storeCount" : 7, "hlogFileCount" : 1, "hlogFileSize" : 0, "storeFileCount" : 2, "memStoreSize" : 5973, "storeFileSize" : 10888, "maxStoreFileAge" : 743, "minStoreFileAge" : 736, "avgStoreFileAge" : 739, "numReferenceFiles" : 0, "regionServerStartTime" : 1529623390382, "averageRegionSize" : 2408, "storeFileIndexSize" : 648, "staticIndexSize" : 75, "staticBloomSize" : 24, "mutationsWithoutWALCount" : 0, "mutationsWithoutWALSize" : 0, "percentFilesLocal" : 0.0, "percentFilesLocalSecondaryRegions" : 0.0, "splitQueueLength" : 0, "compactionQueueLength" : 0, "smallCompactionQueueLength" : 0, "largeCompactionQueueLength" : 0, "flushQueueLength" : 0, "blockCacheFreeSize" : 1043193472, "blockCacheCount" : 5, "blockCacheSize" : 768832, "blockCacheCountHitPercent" : 91.93548387096774, "blockCacheExpressHitPercent" : 91.93548387096774, "l1CacheHitCount" : 57, "l1CacheMissCount" : 5, "l1CacheHitRatio" : 0.9193548387096774, "l1CacheMissRatio" : 0.08064516129032258, "l2CacheHitCount" : 0, "l2CacheMissCount" : 0, "l2CacheHitRatio" : 0.0, "l2CacheMissRatio" : 0.0, "mobFileCacheCount" : 0, "mobFileCacheHitPercent" : 0.0, "totalRequestCount" : 69, "totalRowActionRequestCount" : 259, "readRequestCount" : 104, "filteredReadRequestCount" : 0, "writeRequestCount" : 155, "rpcGetRequestCount" : 55, "rpcScanRequestCount" : 1, "rpcMultiRequestCount" : 6, "rpcMutateRequestCount" : 2, "checkMutateFailedCount" : 0, "checkMutatePassedCount" : 0, "blockCacheHitCount" : 57, "blockCacheHitCountPrimary" : 57, "blockCacheMissCount" : 5, "blockCacheMissCountPrimary" : 5, "blockCacheEvictionCount" : 0, "blockCacheEvictionCountPrimary" : 0, "blockCacheFailedInsertionCount" : 0, "blockCacheDataMissCount" : 0, "blockCacheLeafIndexMissCount" : 0, "blockCacheBloomChunkMissCount" : 5, "blockCacheMetaMissCount" : 0, "blockCacheRootIndexMissCount" : 0, "blockCacheIntermediateIndexMissCount" : 0, "blockCacheFileInfoMissCount" : 0, "blockCacheGeneralBloomMetaMissCount" : 0, "blockCacheDeleteFamilyBloomMissCount" : 0, "blockCacheTrailerMissCount" : 0, "blockCacheDataHitCount" : 0, "blockCacheLeafIndexHitCount" : 0, "blockCacheBloomChunkHitCount" : 57, "blockCacheMetaHitCount" : 0, "blockCacheRootIndexHitCount" : 0, "blockCacheIntermediateIndexHitCount" : 0, "blockCacheFileInfoHitCount" : 0, "blockCacheGeneralBloomMetaHitCount" : 0, "blockCacheDeleteFamilyBloomHitCount" : 0, "blockCacheTrailerHitCount" : 0, "updatesBlockedTime" : 0, "flushedCellsCount" : 32, "compactedCellsCount" : 0, "majorCompactedCellsCount" : 0, "flushedCellsSize" : 1524, "compactedCellsSize" : 0, "majorCompactedCellsSize" : 0, "cellsCountCompactedFromMob" : 0, "cellsCountCompactedToMob" : 0, "cellsSizeCompactedFromMob" : 0, "cellsSizeCompactedToMob" : 0, "mobFlushCount" : 0, "mobFlushedCellsCount" : 0, "mobFlushedCellsSize" : 0, "mobScanCellsCount" : 0, "mobScanCellsSize" : 0, "mobFileCacheAccessCount" : 0, "mobFileCacheMissCount" : 0, "mobFileCacheEvictedCount" : 0, "hedgedReads" : 0, "hedgedReadWins" : 0, "blockedRequestCount" : 0, "MajorCompactionTime_num_ops" : 0, "MajorCompactionTime_min" : 0, "MajorCompactionTime_max" : 0, "MajorCompactionTime_mean" : 0, "MajorCompactionTime_25th_percentile" : 0, "MajorCompactionTime_median" : 0, "MajorCompactionTime_75th_percentile" : 0, "MajorCompactionTime_90th_percentile" : 0, "MajorCompactionTime_95th_percentile" : 0, "MajorCompactionTime_98th_percentile" : 0, "MajorCompactionTime_99th_percentile" : 0, "MajorCompactionTime_99.9th_percentile" : 0, "PauseTimeWithGc_num_ops" : 0, "PauseTimeWithGc_min" : 0, "PauseTimeWithGc_max" : 0, "PauseTimeWithGc_mean" : 0, "PauseTimeWithGc_25th_percentile" : 0, "PauseTimeWithGc_median" : 0, "PauseTimeWithGc_75th_percentile" : 0, "PauseTimeWithGc_90th_percentile" : 0, "PauseTimeWithGc_95th_percentile" : 0, "PauseTimeWithGc_98th_percentile" : 0, "PauseTimeWithGc_99th_percentile" : 0, "PauseTimeWithGc_99.9th_percentile" : 0, "compactedOutputBytes" : 0, "pauseWarnThresholdExceeded" : 0, "ScanTime_num_ops" : 1, "ScanTime_min" : 0, "ScanTime_max" : 0, "ScanTime_mean" : 0, "ScanTime_25th_percentile" : 0, "ScanTime_median" : 0, "ScanTime_75th_percentile" : 0, "ScanTime_90th_percentile" : 0, "ScanTime_95th_percentile" : 0, "ScanTime_98th_percentile" : 0, "ScanTime_99th_percentile" : 0, "ScanTime_99.9th_percentile" : 0, "ScanTime_TimeRangeCount_600000-inf" : 1, "Increment_num_ops" : 0, "Increment_min" : 0, "Increment_max" : 0, "Increment_mean" : 0, "Increment_25th_percentile" : 0, "Increment_median" : 0, "Increment_75th_percentile" : 0, "Increment_90th_percentile" : 0, "Increment_95th_percentile" : 0, "Increment_98th_percentile" : 0, "Increment_99th_percentile" : 0, "Increment_99.9th_percentile" : 0, "Delete_num_ops" : 0, "Delete_min" : 0, "Delete_max" : 0, "Delete_mean" : 0, "Delete_25th_percentile" : 0, "Delete_median" : 0, "Delete_75th_percentile" : 0, "Delete_90th_percentile" : 0, "Delete_95th_percentile" : 0, "Delete_98th_percentile" : 0, "Delete_99th_percentile" : 0, "Delete_99.9th_percentile" : 0, "Put_num_ops" : 2, "Put_min" : 0, "Put_max" : 0, "Put_mean" : 0, "Put_25th_percentile" : 0, "Put_median" : 0, "Put_75th_percentile" : 0, "Put_90th_percentile" : 0, "Put_95th_percentile" : 0, "Put_98th_percentile" : 0, "Put_99th_percentile" : 0, "Put_99.9th_percentile" : 0, "Put_TimeRangeCount_600000-inf" : 2, "DeleteBatch_num_ops" : 0, "DeleteBatch_min" : 0, "DeleteBatch_max" : 0, "DeleteBatch_mean" : 0, "DeleteBatch_25th_percentile" : 0, "DeleteBatch_median" : 0, "DeleteBatch_75th_percentile" : 0, "DeleteBatch_90th_percentile" : 0, "DeleteBatch_95th_percentile" : 0, "DeleteBatch_98th_percentile" : 0, "DeleteBatch_99th_percentile" : 0, "DeleteBatch_99.9th_percentile" : 0, "splitRequestCount" : 0, "FlushMemstoreSize_num_ops" : 2, "FlushMemstoreSize_min" : 0, "FlushMemstoreSize_max" : 0, "FlushMemstoreSize_mean" : 0, "FlushMemstoreSize_25th_percentile" : 0, "FlushMemstoreSize_median" : 0, "FlushMemstoreSize_75th_percentile" : 0, "FlushMemstoreSize_90th_percentile" : 0, "FlushMemstoreSize_95th_percentile" : 0, "FlushMemstoreSize_98th_percentile" : 0, "FlushMemstoreSize_99th_percentile" : 0, "FlushMemstoreSize_99.9th_percentile" : 0, "FlushMemstoreSize_SizeRangeCount_100000000-inf" : 2, "CompactionInputFileCount_num_ops" : 0, "CompactionInputFileCount_min" : 0, "CompactionInputFileCount_max" : 0, "CompactionInputFileCount_mean" : 0, "CompactionInputFileCount_25th_percentile" : 0, "CompactionInputFileCount_median" : 0, "CompactionInputFileCount_75th_percentile" : 0, "CompactionInputFileCount_90th_percentile" : 0, "CompactionInputFileCount_95th_percentile" : 0, "CompactionInputFileCount_98th_percentile" : 0, "CompactionInputFileCount_99th_percentile" : 0, "CompactionInputFileCount_99.9th_percentile" : 0, "PutBatch_num_ops" : 14, "PutBatch_min" : 3, "PutBatch_max" : 15, "PutBatch_mean" : 9, "PutBatch_25th_percentile" : 3, "PutBatch_median" : 4, "PutBatch_75th_percentile" : 15, "PutBatch_90th_percentile" : 15, "PutBatch_95th_percentile" : 15, "PutBatch_98th_percentile" : 15, "PutBatch_99th_percentile" : 15, "PutBatch_99.9th_percentile" : 15, "PutBatch_TimeRangeCount_1-3" : 1, "PutBatch_TimeRangeCount_10-30" : 1, "PutBatch_TimeRangeCount_600000-inf" : 12, "CompactionTime_num_ops" : 0, "CompactionTime_min" : 0, "CompactionTime_max" : 0, "CompactionTime_mean" : 0, "CompactionTime_25th_percentile" : 0, "CompactionTime_median" : 0, "CompactionTime_75th_percentile" : 0, "CompactionTime_90th_percentile" : 0, "CompactionTime_95th_percentile" : 0, "CompactionTime_98th_percentile" : 0, "CompactionTime_99th_percentile" : 0, "CompactionTime_99.9th_percentile" : 0, "Get_num_ops" : 177, "Get_min" : 0, "Get_max" : 31, "Get_mean" : 0, "Get_25th_percentile" : 0, "Get_median" : 0, "Get_75th_percentile" : 0, "Get_90th_percentile" : 1, "Get_95th_percentile" : 4, "Get_98th_percentile" : 7, "Get_99th_percentile" : 31, "Get_99.9th_percentile" : 31, "Get_TimeRangeCount_0-1" : 67, "Get_TimeRangeCount_3-10" : 3, "Get_TimeRangeCount_30-100" : 1, "Get_TimeRangeCount_600000-inf" : 106, "MajorCompactionInputFileCount_num_ops" : 0, "MajorCompactionInputFileCount_min" : 0, "MajorCompactionInputFileCount_max" : 0, "MajorCompactionInputFileCount_mean" : 0, "MajorCompactionInputFileCount_25th_percentile" : 0, "MajorCompactionInputFileCount_median" : 0, "MajorCompactionInputFileCount_75th_percentile" : 0, "MajorCompactionInputFileCount_90th_percentile" : 0, "MajorCompactionInputFileCount_95th_percentile" : 0, "MajorCompactionInputFileCount_98th_percentile" : 0, "MajorCompactionInputFileCount_99th_percentile" : 0, "MajorCompactionInputFileCount_99.9th_percentile" : 0, "CheckAndPut_num_ops" : 0, "CheckAndPut_min" : 0, "CheckAndPut_max" : 0, "CheckAndPut_mean" : 0, "CheckAndPut_25th_percentile" : 0, "CheckAndPut_median" : 0, "CheckAndPut_75th_percentile" : 0, "CheckAndPut_90th_percentile" : 0, "CheckAndPut_95th_percentile" : 0, "CheckAndPut_98th_percentile" : 0, "CheckAndPut_99th_percentile" : 0, "CheckAndPut_99.9th_percentile" : 0, "SplitTime_num_ops" : 0, "SplitTime_min" : 0, "SplitTime_max" : 0, "SplitTime_mean" : 0, "SplitTime_25th_percentile" : 0, "SplitTime_median" : 0, "SplitTime_75th_percentile" : 0, "SplitTime_90th_percentile" : 0, "SplitTime_95th_percentile" : 0, "SplitTime_98th_percentile" : 0, "SplitTime_99th_percentile" : 0, "SplitTime_99.9th_percentile" : 0, "MajorCompactionOutputSize_num_ops" : 0, "MajorCompactionOutputSize_min" : 0, "MajorCompactionOutputSize_max" : 0, "MajorCompactionOutputSize_mean" : 0, "MajorCompactionOutputSize_25th_percentile" : 0, "MajorCompactionOutputSize_median" : 0, "MajorCompactionOutputSize_75th_percentile" : 0, "MajorCompactionOutputSize_90th_percentile" : 0, "MajorCompactionOutputSize_95th_percentile" : 0, "MajorCompactionOutputSize_98th_percentile" : 0, "MajorCompactionOutputSize_99th_percentile" : 0, "MajorCompactionOutputSize_99.9th_percentile" : 0, "majorCompactedInputBytes" : 0, "slowAppendCount" : 0, "flushedOutputBytes" : 10888, "CompactionOutputFileCount_num_ops" : 0, "CompactionOutputFileCount_min" : 0, "CompactionOutputFileCount_max" : 0, "CompactionOutputFileCount_mean" : 0, "CompactionOutputFileCount_25th_percentile" : 0, "CompactionOutputFileCount_median" : 0, "CompactionOutputFileCount_75th_percentile" : 0, "CompactionOutputFileCount_90th_percentile" : 0, "CompactionOutputFileCount_95th_percentile" : 0, "CompactionOutputFileCount_98th_percentile" : 0, "CompactionOutputFileCount_99th_percentile" : 0, "CompactionOutputFileCount_99.9th_percentile" : 0, "slowDeleteCount" : 0, "Replay_num_ops" : 0, "Replay_min" : 0, "Replay_max" : 0, "Replay_mean" : 0, "Replay_25th_percentile" : 0, "Replay_median" : 0, "Replay_75th_percentile" : 0, "Replay_90th_percentile" : 0, "Replay_95th_percentile" : 0, "Replay_98th_percentile" : 0, "Replay_99th_percentile" : 0, "Replay_99.9th_percentile" : 0, "FlushTime_num_ops" : 2, "FlushTime_min" : 0, "FlushTime_max" : 0, "FlushTime_mean" : 0, "FlushTime_25th_percentile" : 0, "FlushTime_median" : 0, "FlushTime_75th_percentile" : 0, "FlushTime_90th_percentile" : 0, "FlushTime_95th_percentile" : 0, "FlushTime_98th_percentile" : 0, "FlushTime_99th_percentile" : 0, "FlushTime_99.9th_percentile" : 0, "FlushTime_TimeRangeCount_600000-inf" : 2, "MajorCompactionInputSize_num_ops" : 0, "MajorCompactionInputSize_min" : 0, "MajorCompactionInputSize_max" : 0, "MajorCompactionInputSize_mean" : 0, "MajorCompactionInputSize_25th_percentile" : 0, "MajorCompactionInputSize_median" : 0, "MajorCompactionInputSize_75th_percentile" : 0, "MajorCompactionInputSize_90th_percentile" : 0, "MajorCompactionInputSize_95th_percentile" : 0, "MajorCompactionInputSize_98th_percentile" : 0, "MajorCompactionInputSize_99th_percentile" : 0, "MajorCompactionInputSize_99.9th_percentile" : 0, "pauseInfoThresholdExceeded" : 0, "splitSuccessCount" : 0, "CheckAndDelete_num_ops" : 0, "CheckAndDelete_min" : 0, "CheckAndDelete_max" : 0, "CheckAndDelete_mean" : 0, "CheckAndDelete_25th_percentile" : 0, "CheckAndDelete_median" : 0, "CheckAndDelete_75th_percentile" : 0, "CheckAndDelete_90th_percentile" : 0, "CheckAndDelete_95th_percentile" : 0, "CheckAndDelete_98th_percentile" : 0, "CheckAndDelete_99th_percentile" : 0, "CheckAndDelete_99.9th_percentile" : 0, "CompactionInputSize_num_ops" : 0, "CompactionInputSize_min" : 0, "CompactionInputSize_max" : 0, "CompactionInputSize_mean" : 0, "CompactionInputSize_25th_percentile" : 0, "CompactionInputSize_median" : 0, "CompactionInputSize_75th_percentile" : 0, "CompactionInputSize_90th_percentile" : 0, "CompactionInputSize_95th_percentile" : 0, "CompactionInputSize_98th_percentile" : 0, "CompactionInputSize_99th_percentile" : 0, "CompactionInputSize_99.9th_percentile" : 0, "MajorCompactionOutputFileCount_num_ops" : 0, "MajorCompactionOutputFileCount_min" : 0, "MajorCompactionOutputFileCount_max" : 0, "MajorCompactionOutputFileCount_mean" : 0, "MajorCompactionOutputFileCount_25th_percentile" : 0, "MajorCompactionOutputFileCount_median" : 0, "MajorCompactionOutputFileCount_75th_percentile" : 0, "MajorCompactionOutputFileCount_90th_percentile" : 0, "MajorCompactionOutputFileCount_95th_percentile" : 0, "MajorCompactionOutputFileCount_98th_percentile" : 0, "MajorCompactionOutputFileCount_99th_percentile" : 0, "MajorCompactionOutputFileCount_99.9th_percentile" : 0, "ScanSize_num_ops" : 1, "ScanSize_min" : 0, "ScanSize_max" : 0, "ScanSize_mean" : 0, "ScanSize_25th_percentile" : 0, "ScanSize_median" : 0, "ScanSize_75th_percentile" : 0, "ScanSize_90th_percentile" : 0, "ScanSize_95th_percentile" : 0, "ScanSize_98th_percentile" : 0, "ScanSize_99th_percentile" : 0, "ScanSize_99.9th_percentile" : 0, "ScanSize_SizeRangeCount_100000000-inf" : 1, "slowGetCount" : 0, "flushedMemstoreBytes" : 1524, "CompactionOutputSize_num_ops" : 0, "CompactionOutputSize_min" : 0, "CompactionOutputSize_max" : 0, "CompactionOutputSize_mean" : 0, "CompactionOutputSize_25th_percentile" : 0, "CompactionOutputSize_median" : 0, "CompactionOutputSize_75th_percentile" : 0, "CompactionOutputSize_90th_percentile" : 0, "CompactionOutputSize_95th_percentile" : 0, "CompactionOutputSize_98th_percentile" : 0, "CompactionOutputSize_99th_percentile" : 0, "CompactionOutputSize_99.9th_percentile" : 0, "majorCompactedOutputBytes" : 0, "PauseTimeWithoutGc_num_ops" : 0, "PauseTimeWithoutGc_min" : 0, "PauseTimeWithoutGc_max" : 0, "PauseTimeWithoutGc_mean" : 0, "PauseTimeWithoutGc_25th_percentile" : 0, "PauseTimeWithoutGc_median" : 0, "PauseTimeWithoutGc_75th_percentile" : 0, "PauseTimeWithoutGc_90th_percentile" : 0, "PauseTimeWithoutGc_95th_percentile" : 0, "PauseTimeWithoutGc_98th_percentile" : 0, "PauseTimeWithoutGc_99th_percentile" : 0, "PauseTimeWithoutGc_99.9th_percentile" : 0, "slowPutCount" : 0, "slowIncrementCount" : 0, "compactedInputBytes" : 0, "Append_num_ops" : 0, "Append_min" : 0, "Append_max" : 0, "Append_mean" : 0, "Append_25th_percentile" : 0, "Append_median" : 0, "Append_75th_percentile" : 0, "Append_90th_percentile" : 0, "Append_95th_percentile" : 0, "Append_98th_percentile" : 0, "Append_99th_percentile" : 0, "Append_99.9th_percentile" : 0, "FlushOutputSize_num_ops" : 2, "FlushOutputSize_min" : 0, "FlushOutputSize_max" : 0, "FlushOutputSize_mean" : 0, "FlushOutputSize_25th_percentile" : 0, "FlushOutputSize_median" : 0, "FlushOutputSize_75th_percentile" : 0, "FlushOutputSize_90th_percentile" : 0, "FlushOutputSize_95th_percentile" : 0, "FlushOutputSize_98th_percentile" : 0, "FlushOutputSize_99th_percentile" : 0, "FlushOutputSize_99.9th_percentile" : 0, "FlushOutputSize_SizeRangeCount_100000000-inf" : 2, "Bulkload_count" : 0, "Bulkload_mean_rate" : 0.0, "Bulkload_1min_rate" : 0.0, "Bulkload_5min_rate" : 0.0, "Bulkload_15min_rate" : 0.0, "Bulkload_num_ops" : 0, "Bulkload_min" : 0, "Bulkload_max" : 0, "Bulkload_mean" : 0, "Bulkload_25th_percentile" : 0, "Bulkload_median" : 0, "Bulkload_75th_percentile" : 0, "Bulkload_90th_percentile" : 0, "Bulkload_95th_percentile" : 0, "Bulkload_98th_percentile" : 0, "Bulkload_99th_percentile" : 0, "Bulkload_99.9th_percentile" : 0 } ] } 2018-06-21 23:23:34,296 ERROR [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=55146] master.MasterRpcServices(509): Region server asf911.gq1.ygridcore.net,40767,1529623390148 reported a fatal error: ***** ABORTING region server asf911.gq1.ygridcore.net,40767,1529623390148: Aborting for tests ***** Cause: java.lang.Exception: doTestFlushCommits at org.apache.hadoop.hbase.client.TestMultiParallel.doTestFlushCommits(TestMultiParallel.java:312) at org.apache.hadoop.hbase.client.TestMultiParallel.testFlushCommitsWithAbort(TestMultiParallel.java:289) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) 2018-06-21 23:23:34,299 INFO [Time-limited test] regionserver.HRegionServer(2148): ***** STOPPING region server 'asf911.gq1.ygridcore.net,40767,1529623390148' ***** 2018-06-21 23:23:34,299 INFO [Time-limited test] regionserver.HRegionServer(2162): STOPPED: Aborting for tests 2018-06-21 23:23:34,299 INFO [RS:1;asf911:40767] regionserver.SplitLogWorker(166): Sending interrupt to stop the worker thread 2018-06-21 23:23:34,299 INFO [RS:1;asf911:40767] regionserver.HeapMemoryManager(221): Stopping 2018-06-21 23:23:34,300 INFO [RS:1;asf911:40767] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager abruptly. 2018-06-21 23:23:34,316 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(383): MemStoreFlusher.1 exiting 2018-06-21 23:23:34,316 INFO [SplitLogWorker-asf911:40767] regionserver.SplitLogWorker(148): SplitLogWorker interrupted. Exiting. 2018-06-21 23:23:34,316 INFO [SplitLogWorker-asf911:40767] regionserver.SplitLogWorker(157): SplitLogWorker asf911.gq1.ygridcore.net,40767,1529623390148 exiting 2018-06-21 23:23:34,316 INFO [RS:1;asf911:40767] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager abruptly. 2018-06-21 23:23:34,300 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(383): MemStoreFlusher.0 exiting 2018-06-21 23:23:34,322 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1520): Closing d4ac3b44d02d68507d6317e201672011, disabling compactions & flushes 2018-06-21 23:23:34,322 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1560): Updates disabled for region multi_test_table,sss,1529623399386.d4ac3b44d02d68507d6317e201672011. 2018-06-21 23:23:34,322 INFO [RS:1;asf911:40767] regionserver.HRegionServer(1066): aborting server asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:34,322 DEBUG [RS:1;asf911:40767] zookeeper.MetaTableLocator(642): Stopping MetaTableLocator 2018-06-21 23:23:34,322 INFO [RS:1;asf911:40767] zookeeper.ReadOnlyZKClient(350): Close zookeeper connection 0x57f0e7d5 to localhost:58989 2018-06-21 23:23:34,322 DEBUG [RS:1;asf911:40767] ipc.AbstractRpcClient(483): Stopping rpc client 2018-06-21 23:23:34,322 ERROR [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1661): Memstore data size is 2184 2018-06-21 23:23:34,323 INFO [RS:1;asf911:40767] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-06-21 23:23:34,323 INFO [RS:1;asf911:40767] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-06-21 23:23:34,323 INFO [RS:1;asf911:40767] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-06-21 23:23:34,324 INFO [RS:1;asf911:40767] regionserver.HRegionServer(1399): Waiting on 8 regions to close 2018-06-21 23:23:34,324 DEBUG [RS:1;asf911:40767] regionserver.HRegionServer(1403): Online Regions={d4ac3b44d02d68507d6317e201672011=multi_test_table,sss,1529623399386.d4ac3b44d02d68507d6317e201672011., 40503e5e9a757a4ccf579bf29b8d78c6=multi_test_table,uuu,1529623399386.40503e5e9a757a4ccf579bf29b8d78c6., c793c2c3948aad84865d30ebf71a529e=multi_test_table,eee,1529623399386.c793c2c3948aad84865d30ebf71a529e., 91ab16f39c84b448bdc2e5a1e1fc801b=multi_test_table,hhh,1529623399386.91ab16f39c84b448bdc2e5a1e1fc801b., 1588230740=hbase:meta,,1.1588230740, 0dfb3ebb8c759d632ec793ac0387ec95=multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95., 36d0a953419e9f81636aebf73e7ed5df=multi_test_table,ppp,1529623399386.36d0a953419e9f81636aebf73e7ed5df., fee67ae5d4f2b2be35fa516877a8d173=multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173.} 2018-06-21 23:23:34,326 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1520): Closing c793c2c3948aad84865d30ebf71a529e, disabling compactions & flushes 2018-06-21 23:23:34,330 DEBUG [RS_CLOSE_META-regionserver/asf911:0-0] regionserver.HRegion(1520): Closing 1588230740, disabling compactions & flushes 2018-06-21 23:23:34,330 DEBUG [RS_CLOSE_META-regionserver/asf911:0-0] regionserver.HRegion(1560): Updates disabled for region hbase:meta,,1.1588230740 2018-06-21 23:23:34,327 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1674): Closed multi_test_table,sss,1529623399386.d4ac3b44d02d68507d6317e201672011. 2018-06-21 23:23:34,330 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1520): Closing 40503e5e9a757a4ccf579bf29b8d78c6, disabling compactions & flushes 2018-06-21 23:23:34,330 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1560): Updates disabled for region multi_test_table,eee,1529623399386.c793c2c3948aad84865d30ebf71a529e. 2018-06-21 23:23:34,331 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1560): Updates disabled for region multi_test_table,uuu,1529623399386.40503e5e9a757a4ccf579bf29b8d78c6. 2018-06-21 23:23:34,331 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] handler.CloseRegionHandler(124): Closed multi_test_table,sss,1529623399386.d4ac3b44d02d68507d6317e201672011. 2018-06-21 23:23:34,332 ERROR [RS_CLOSE_META-regionserver/asf911:0-0] regionserver.HRegion(1661): Memstore data size is 48069 2018-06-21 23:23:34,337 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1520): Closing 91ab16f39c84b448bdc2e5a1e1fc801b, disabling compactions & flushes 2018-06-21 23:23:34,337 DEBUG [RS_CLOSE_META-regionserver/asf911:0-0] coprocessor.CoprocessorHost(288): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-06-21 23:23:34,337 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1560): Updates disabled for region multi_test_table,hhh,1529623399386.91ab16f39c84b448bdc2e5a1e1fc801b. 2018-06-21 23:23:34,337 ERROR [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1661): Memstore data size is 2340 2018-06-21 23:23:34,338 ERROR [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1661): Memstore data size is 2340 2018-06-21 23:23:34,341 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1674): Closed multi_test_table,hhh,1529623399386.91ab16f39c84b448bdc2e5a1e1fc801b. 2018-06-21 23:23:34,341 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1674): Closed multi_test_table,eee,1529623399386.c793c2c3948aad84865d30ebf71a529e. 2018-06-21 23:23:34,341 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] handler.CloseRegionHandler(124): Closed multi_test_table,hhh,1529623399386.91ab16f39c84b448bdc2e5a1e1fc801b. 2018-06-21 23:23:34,341 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] handler.CloseRegionHandler(124): Closed multi_test_table,eee,1529623399386.c793c2c3948aad84865d30ebf71a529e. 2018-06-21 23:23:34,341 ERROR [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1661): Memstore data size is 2184 2018-06-21 23:23:34,341 INFO [RS_CLOSE_META-regionserver/asf911:0-0] regionserver.HRegion(1674): Closed hbase:meta,,1.1588230740 2018-06-21 23:23:34,342 DEBUG [RS_CLOSE_META-regionserver/asf911:0-0] handler.CloseRegionHandler(124): Closed hbase:meta,,1.1588230740 2018-06-21 23:23:34,343 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1520): Closing 0dfb3ebb8c759d632ec793ac0387ec95, disabling compactions & flushes 2018-06-21 23:23:34,343 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1520): Closing 36d0a953419e9f81636aebf73e7ed5df, disabling compactions & flushes 2018-06-21 23:23:34,343 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1674): Closed multi_test_table,uuu,1529623399386.40503e5e9a757a4ccf579bf29b8d78c6. 2018-06-21 23:23:34,344 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] handler.CloseRegionHandler(124): Closed multi_test_table,uuu,1529623399386.40503e5e9a757a4ccf579bf29b8d78c6. 2018-06-21 23:23:34,343 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1560): Updates disabled for region multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95. 2018-06-21 23:23:34,345 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1520): Closing fee67ae5d4f2b2be35fa516877a8d173, disabling compactions & flushes 2018-06-21 23:23:34,344 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1560): Updates disabled for region multi_test_table,ppp,1529623399386.36d0a953419e9f81636aebf73e7ed5df. 2018-06-21 23:23:34,345 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1560): Updates disabled for region multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173. 2018-06-21 23:23:34,353 ERROR [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1661): Memstore data size is 2184 2018-06-21 23:23:34,353 ERROR [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1661): Memstore data size is 1092 2018-06-21 23:23:34,354 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1674): Closed multi_test_table,ppp,1529623399386.36d0a953419e9f81636aebf73e7ed5df. 2018-06-21 23:23:34,356 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1674): Closed multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95. 2018-06-21 23:23:34,356 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] handler.CloseRegionHandler(124): Closed multi_test_table,ppp,1529623399386.36d0a953419e9f81636aebf73e7ed5df. 2018-06-21 23:23:34,357 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] handler.CloseRegionHandler(124): Closed multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95. 2018-06-21 23:23:34,358 ERROR [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1661): Memstore data size is 546 2018-06-21 23:23:34,363 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1674): Closed multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173. 2018-06-21 23:23:34,363 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] handler.CloseRegionHandler(124): Closed multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173. 2018-06-21 23:23:34,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=40767] ipc.CallRunner(142): callId: 467 service: ClientService methodName: Multi size: 7.5 K connection: 67.195.81.155:45610 deadline: 1529623474424, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server asf911.gq1.ygridcore.net,40767,1529623390148 aborting 2018-06-21 23:23:34,524 INFO [RS:1;asf911:40767] regionserver.HRegionServer(1097): stopping server asf911.gq1.ygridcore.net,40767,1529623390148; all regions closed. 2018-06-21 23:23:34,531 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=2,queue=0,port=40767] ipc.CallRunner(142): callId: 470 service: ClientService methodName: Scan size: 146 connection: 67.195.81.155:45610 deadline: 1529623474530, exception=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server asf911.gq1.ygridcore.net,40767,1529623390148 aborting 2018-06-21 23:23:34,531 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741836_1012{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:34,531 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741836_1012{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:34,532 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741836_1012 size 53146 2018-06-21 23:23:34,542 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741835_1011{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:34,542 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741835_1011{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:34,542 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741835_1011{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:34,543 DEBUG [RS:1;asf911:40767] ipc.AbstractRpcClient(483): Stopping rpc client 2018-06-21 23:23:34,544 INFO [RS:1;asf911:40767] regionserver.Leases(149): Closed leases 2018-06-21 23:23:34,544 INFO [RS:1;asf911:40767] hbase.ChoreService(327): Chore service for: regionserver/asf911:0 had [[ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region asf911.gq1.ygridcore.net,40767,1529623390148 Period: 120000 Unit: MILLISECONDS]] on shutdown 2018-06-21 23:23:34,546 INFO [regionserver/asf911:0.logRoller] regionserver.LogRoller(212): LogRoller exiting. 2018-06-21 23:23:34,548 INFO [RS:1;asf911:40767] ipc.NettyRpcServer(143): Stopping server on /67.195.81.155:40767 2018-06-21 23:23:34,583 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:34,583 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:34,583 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:34,583 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:34,583 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:34,584 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 2018-06-21 23:23:34,584 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:40767-0x16424a89dd40002, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:34,591 INFO [RS:1;asf911:40767] regionserver.HRegionServer(1153): Exiting; stopping=asf911.gq1.ygridcore.net,40767,1529623390148; zookeeper connection closed. 2018-06-21 23:23:34,592 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:34,592 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@b21daa3] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(221): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@b21daa3 2018-06-21 23:23:34,592 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:34,592 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:34,592 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:34,592 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:34,592 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:34,593 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:34,593 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:34,593 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:34,593 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:34,593 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:34,593 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:34,593 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:34,593 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:34,594 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:34,594 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:34,594 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:34,594 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:34,594 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:34,594 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:34,594 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:34,594 INFO [Time-limited test-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(122): /hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 znode expired, triggering replicatorRemoved event 2018-06-21 23:23:34,595 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:34,595 INFO [Time-limited test-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(122): /hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 znode expired, triggering replicatorRemoved event 2018-06-21 23:23:34,594 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:34,595 INFO [Time-limited test-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(122): /hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 znode expired, triggering replicatorRemoved event 2018-06-21 23:23:34,595 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:34,595 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:34,596 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:34,596 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:34,596 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:34,597 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:34,597 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:34,595 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:34,595 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:34,599 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:34,599 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:34,600 INFO [RegionServerTracker-0] master.RegionServerTracker(159): RegionServer ephemeral node deleted, processing expiration [asf911.gq1.ygridcore.net,40767,1529623390148] 2018-06-21 23:23:34,600 INFO [RegionServerTracker-0] master.ServerManager(604): Processing expiration of asf911.gq1.ygridcore.net,40767,1529623390148 on asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:34,601 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:34,601 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:34,595 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:34,599 INFO [Time-limited test-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(122): /hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 znode expired, triggering replicatorRemoved event 2018-06-21 23:23:34,602 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:34,602 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:34,603 INFO [regionserver/asf911:0.leaseChecker] regionserver.Leases(149): Closed leases 2018-06-21 23:23:34,602 INFO [Time-limited test-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(122): /hbase/rs/asf911.gq1.ygridcore.net,40767,1529623390148 znode expired, triggering replicatorRemoved event 2018-06-21 23:23:34,603 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:34,603 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:34,603 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:34,604 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:34,604 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:34,604 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:34,604 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:34,604 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:34,605 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:34,605 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:34,605 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:34,605 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:34,605 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:34,605 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:34,605 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:34,606 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:34,639 DEBUG [Default-IPC-NioEventLoopGroup-9-16] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:40767 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:40767 2018-06-21 23:23:34,804 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(887): Stored pid=55, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure server=asf911.gq1.ygridcore.net,40767,1529623390148, splitWal=true, meta=true 2018-06-21 23:23:34,804 DEBUG [RegionServerTracker-0] assignment.AssignmentManager(1334): Added=asf911.gq1.ygridcore.net,40767,1529623390148 to dead servers, submitted shutdown handler to be executed meta=true 2018-06-21 23:23:34,806 INFO [PEWorker-9] procedure.ServerCrashProcedure(118): Start pid=55, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure server=asf911.gq1.ygridcore.net,40767,1529623390148, splitWal=true, meta=true 2018-06-21 23:23:34,844 DEBUG [hconnection-0xe1c5692-metaLookup-shared--pool16-t32] ipc.AbstractRpcClient(349): Not trying to connect to asf911.gq1.ygridcore.net/67.195.81.155:40767 this server is in the failed servers list 2018-06-21 23:23:34,906 DEBUG [PEWorker-9] procedure.ServerCrashProcedure(229): Splitting meta WALs pid=55, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS; ServerCrashProcedure server=asf911.gq1.ygridcore.net,40767,1529623390148, splitWal=true, meta=true 2018-06-21 23:23:34,909 DEBUG [PEWorker-9] master.MasterWalManager(283): Renamed region directory: hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,40767,1529623390148-splitting 2018-06-21 23:23:34,909 INFO [PEWorker-9] master.SplitLogManager(461): dead splitlog workers [asf911.gq1.ygridcore.net,40767,1529623390148] 2018-06-21 23:23:34,911 INFO [PEWorker-9] master.SplitLogManager(241): Started splitting 1 logs in [hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,40767,1529623390148-splitting] for [asf911.gq1.ygridcore.net,40767,1529623390148] 2018-06-21 23:23:34,933 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:34,933 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:34,933 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:34,933 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:34,933 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:34,934 DEBUG [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination(332): Put up splitlog task at znode /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.meta.1529623394409.meta 2018-06-21 23:23:34,935 DEBUG [SplitLogWorker-asf911:52507] coordination.ZkSplitLogWorkerCoordination(436): Current region server asf911.gq1.ygridcore.net,52507,1529623390228 is ready to take more tasks, will get task list and try grab tasks again. 2018-06-21 23:23:34,935 DEBUG [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination(364): Task not yet acquired /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.meta.1529623394409.meta, ver=0 2018-06-21 23:23:34,935 DEBUG [SplitLogWorker-asf911:56613] coordination.ZkSplitLogWorkerCoordination(436): Current region server asf911.gq1.ygridcore.net,56613,1529623390297 is ready to take more tasks, will get task list and try grab tasks again. 2018-06-21 23:23:34,935 DEBUG [SplitLogWorker-asf911:34033] coordination.ZkSplitLogWorkerCoordination(436): Current region server asf911.gq1.ygridcore.net,34033,1529623407773 is ready to take more tasks, will get task list and try grab tasks again. 2018-06-21 23:23:34,935 DEBUG [SplitLogWorker-asf911:58235] coordination.ZkSplitLogWorkerCoordination(436): Current region server asf911.gq1.ygridcore.net,58235,1529623390382 is ready to take more tasks, will get task list and try grab tasks again. 2018-06-21 23:23:34,935 DEBUG [SplitLogWorker-asf911:55146] coordination.ZkSplitLogWorkerCoordination(436): Current region server asf911.gq1.ygridcore.net,55146,1529623388534 is ready to take more tasks, will get task list and try grab tasks again. 2018-06-21 23:23:34,949 INFO [SplitLogWorker-asf911:56613] coordination.ZkSplitLogWorkerCoordination(258): worker asf911.gq1.ygridcore.net,56613,1529623390297 acquired task /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.meta.1529623394409.meta 2018-06-21 23:23:34,949 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.meta.1529623394409.meta 2018-06-21 23:23:34,965 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@69ed135] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:42074 to delete [blk_1073741868_1044, blk_1073741869_1045, blk_1073741870_1046] 2018-06-21 23:23:34,972 INFO [RS_LOG_REPLAY_OPS-regionserver/asf911:0-1] wal.WALSplitter(250): Splitting WAL=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,40767,1529623390148-splitting/asf911.gq1.ygridcore.net%2C40767%2C1529623390148.meta.1529623394409.meta, length=53146 2018-06-21 23:23:34,982 INFO [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination(454): Task /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.meta.1529623394409.meta acquired by asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:35,025 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.meta.1529623394409.meta 2018-06-21 23:23:35,025 INFO [RS_LOG_REPLAY_OPS-regionserver/asf911:0-1] util.FSHDFSUtils(195): Recover lease on dfs file hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,40767,1529623390148-splitting/asf911.gq1.ygridcore.net%2C40767%2C1529623390148.meta.1529623394409.meta 2018-06-21 23:23:35,025 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.meta.1529623394409.meta 2018-06-21 23:23:35,026 INFO [RS_LOG_REPLAY_OPS-regionserver/asf911:0-1] util.FSHDFSUtils(284): Recovered lease, attempt=0 on file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,40767,1529623390148-splitting/asf911.gq1.ygridcore.net%2C40767%2C1529623390148.meta.1529623394409.meta after 1ms 2018-06-21 23:23:35,035 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-1] wal.WALSplitter(283): DLS Last flushed sequenceid for 1588230740: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } store_sequence_id { family_name: "rep_barrier" sequence_id: 18446744073709551615 } store_sequence_id { family_name: "table" sequence_id: 4 } 2018-06-21 23:23:35,047 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-1-Writer-2] wal.WALSplitter$LogRecoveredEditsOutputSink(1506): Creating writer path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C40767%2C1529623390148.meta.1529623394409.meta.temp 2018-06-21 23:23:35,055 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-1] wal.WALSplitter(335): Finishing writing output logs and closing down 2018-06-21 23:23:35,055 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-1] wal.WALSplitter$OutputSink(1162): Waiting for split writer threads to finish 2018-06-21 23:23:35,058 INFO [RS_LOG_REPLAY_OPS-regionserver/asf911:0-1] wal.WALSplitter$OutputSink(1186): 3 split writers finished; closing. 2018-06-21 23:23:35,073 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741882_1058{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:35,074 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741882_1058 size 52916 2018-06-21 23:23:35,074 DEBUG [split-log-closeStream-1] wal.WALSplitter$LogRecoveredEditsOutputSink(1381): Closed wap hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C40767%2C1529623390148.meta.1529623394409.meta.temp (wrote 90 edits, skipped 0 edits in 21ms 2018-06-21 23:23:35,075 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741882_1058 size 52916 2018-06-21 23:23:35,077 INFO [split-log-closeStream-1] wal.WALSplitter$LogRecoveredEditsOutputSink(1407): Rename hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C40767%2C1529623390148.meta.1529623394409.meta.temp to hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/recovered.edits/0000000000000000093 2018-06-21 23:23:35,077 INFO [RS_LOG_REPLAY_OPS-regionserver/asf911:0-1] wal.WALSplitter(356): Processed 90 edits across 1 regions; edits skipped=1; log file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,40767,1529623390148-splitting/asf911.gq1.ygridcore.net%2C40767%2C1529623390148.meta.1529623394409.meta, length=53146, corrupted=false, progress failed=false 2018-06-21 23:23:35,113 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.meta.1529623394409.meta 2018-06-21 23:23:35,113 INFO [RS_LOG_REPLAY_OPS-regionserver/asf911:0-1] coordination.ZkSplitLogWorkerCoordination(574): successfully transitioned task /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.meta.1529623394409.meta to final state DONE asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:35,113 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.meta.1529623394409.meta 2018-06-21 23:23:35,114 INFO [RS_LOG_REPLAY_OPS-regionserver/asf911:0-1] handler.WALSplitterHandler(99): Worker asf911.gq1.ygridcore.net,56613,1529623390297 done with task org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$ZkSplitTaskDetails@7677b81d in 164ms. Status = DONE 2018-06-21 23:23:35,114 INFO [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination(372): Task /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.meta.1529623394409.meta entered state=DONE asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:35,120 INFO [Time-limited test-EventThread] wal.WALSplitter(448): Archived processed log hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,40767,1529623390148-splitting/asf911.gq1.ygridcore.net%2C40767%2C1529623390148.meta.1529623394409.meta to hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/oldWALs/asf911.gq1.ygridcore.net%2C40767%2C1529623390148.meta.1529623394409.meta 2018-06-21 23:23:35,121 INFO [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination(415): Done splitting /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.meta.1529623394409.meta 2018-06-21 23:23:35,133 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.meta.1529623394409.meta 2018-06-21 23:23:35,133 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:35,133 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.meta.1529623394409.meta 2018-06-21 23:23:35,134 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:35,134 DEBUG [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination$DeleteAsyncCallback(696): Deleted /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.meta.1529623394409.meta 2018-06-21 23:23:35,133 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:35,133 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:35,133 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:35,135 WARN [PEWorker-9] master.SplitLogManager(279): Returning success without actually splitting and deleting all the log files in path hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,40767,1529623390148-splitting: [FileStatus{path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,40767,1529623390148-splitting/asf911.gq1.ygridcore.net%2C40767%2C1529623390148.1529623393801; isDirectory=false; length=17942; replication=3; blocksize=268435456; modification_time=1529623414543; access_time=1529623393885; owner=jenkins.hfs.1; group=supergroup; permission=rw-r--r--; isSymlink=false}] org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.PathIsNotEmptyDirectoryException): `/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,40767,1529623390148-splitting is non empty': Directory is not empty at org.apache.hadoop.hdfs.server.namenode.FSDirDeleteOp.delete(FSDirDeleteOp.java:84) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:3676) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.delete(NameNodeRpcServer.java:953) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.delete(ClientNamenodeProtocolServerSideTranslatorPB.java:623) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213) at org.apache.hadoop.ipc.Client.call(Client.java:1476) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy28.delete(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:545) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy31.delete(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy32.delete(Unknown Source) at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy32.delete(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2044) at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:707) at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:703) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:714) at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:273) at org.apache.hadoop.hbase.master.MasterWalManager.splitLog(MasterWalManager.java:319) at org.apache.hadoop.hbase.master.MasterWalManager.splitMetaLog(MasterWalManager.java:257) at org.apache.hadoop.hbase.master.MasterWalManager.splitMetaLog(MasterWalManager.java:249) at org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.splitMetaLogs(ServerCrashProcedure.java:233) at org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.executeFromState(ServerCrashProcedure.java:127) at org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.executeFromState(ServerCrashProcedure.java:58) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:189) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:850) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1473) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1241) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:75) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1761) 2018-06-21 23:23:35,135 INFO [PEWorker-9] master.SplitLogManager(293): finished splitting (more than or equal to) 53146 bytes in 1 log files in [hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,40767,1529623390148-splitting] in 224ms 2018-06-21 23:23:35,138 DEBUG [PEWorker-9] procedure.ServerCrashProcedure(235): Done splitting meta WALs pid=55, state=RUNNABLE:SERVER_CRASH_SPLIT_META_LOGS; ServerCrashProcedure server=asf911.gq1.ygridcore.net,40767,1529623390148, splitWal=true, meta=true 2018-06-21 23:23:35,146 DEBUG [hconnection-0xe1c5692-metaLookup-shared--pool16-t32] ipc.AbstractRpcClient(349): Not trying to connect to asf911.gq1.ygridcore.net/67.195.81.155:40767 this server is in the failed servers list 2018-06-21 23:23:35,292 INFO [PEWorker-1] procedure2.ProcedureExecutor(1516): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740}] 2018-06-21 23:23:35,340 INFO [PEWorker-11] procedure.MasterProcedureScheduler(696): pid=56, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740 checking lock on 1588230740 2018-06-21 23:23:35,340 INFO [PEWorker-11] assignment.AssignProcedure(218): Starting pid=56, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=asf911.gq1.ygridcore.net,40767,1529623390148; forceNewPlan=false, retain=true 2018-06-21 23:23:35,491 INFO [master/asf911:0] balancer.BaseLoadBalancer(1497): Reassigned 1 regions. 0 retained the pre-restart assignment. 1 regions were assigned to random hosts, since the old hosts for these regions are no longer present in the cluster. These hosts were: 2018-06-21 23:23:35,492 INFO [PEWorker-4] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:35,500 INFO [PEWorker-4] assignment.RegionTransitionProcedure(241): Dispatch pid=56, ppid=55, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:35,650 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(119): Fallback to compat rpc execution for serverName=asf911.gq1.ygridcore.net,55146,1529623388534 version=0 2018-06-21 23:23:35,651 INFO [RSProcedureDispatcher-pool3-t31] regionserver.RSRpcServices(1980): Open hbase:meta,,1.1588230740 2018-06-21 23:23:35,652 INFO [RS_OPEN_META-master/asf911:0-0] wal.WALFactory(137): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-06-21 23:23:35,656 INFO [RS_OPEN_META-master/asf911:0-0] wal.AbstractFSWAL(424): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C55146%2C1529623388534.meta, suffix=.meta, logDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,55146,1529623388534, archiveDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/oldWALs 2018-06-21 23:23:35,663 DEBUG [RS-EventLoopGroup-10-9] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:53553,DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4,DISK] 2018-06-21 23:23:35,663 DEBUG [RS-EventLoopGroup-10-10] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42074,DS-85f0e371-3357-43c0-95ea-ca66d237b4a8,DISK] 2018-06-21 23:23:35,663 DEBUG [RS-EventLoopGroup-10-11] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:57095,DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d,DISK] 2018-06-21 23:23:35,666 INFO [RS_OPEN_META-master/asf911:0-0] wal.AbstractFSWAL(679): New WAL /user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,55146,1529623388534/asf911.gq1.ygridcore.net%2C55146%2C1529623388534.meta.1529623415657.meta 2018-06-21 23:23:35,666 DEBUG [RS_OPEN_META-master/asf911:0-0] wal.AbstractFSWAL(769): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:53553,DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4,DISK], DatanodeInfoWithStorage[127.0.0.1:42074,DS-85f0e371-3357-43c0-95ea-ca66d237b4a8,DISK], DatanodeInfoWithStorage[127.0.0.1:57095,DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d,DISK]] 2018-06-21 23:23:35,666 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(7094): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2018-06-21 23:23:35,667 DEBUG [RS_OPEN_META-master/asf911:0-0] coprocessor.CoprocessorHost(200): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2018-06-21 23:23:35,667 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(8072): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2018-06-21 23:23:35,669 INFO [RS_OPEN_META-master/asf911:0-0] regionserver.RegionCoprocessorHost(394): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2018-06-21 23:23:35,670 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table meta 1588230740 2018-06-21 23:23:35,670 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(827): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:35,670 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(7134): checking encryption for 1588230740 2018-06-21 23:23:35,670 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(7139): checking classloading for 1588230740 2018-06-21 23:23:35,699 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/info 2018-06-21 23:23:35,699 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/info 2018-06-21 23:23:35,700 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=5, currentSize=750.81 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=750.81 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:35,700 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:35,701 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:35,703 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/rep_barrier 2018-06-21 23:23:35,703 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/rep_barrier 2018-06-21 23:23:35,704 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=5, currentSize=750.81 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=750.81 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:35,704 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:35,705 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:35,707 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/table 2018-06-21 23:23:35,708 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/table 2018-06-21 23:23:35,708 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=5, currentSize=750.81 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=750.81 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:35,709 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:35,710 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:35,710 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(923): replaying wal for 1588230740 2018-06-21 23:23:35,713 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(4475): Found 1 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740 2018-06-21 23:23:35,715 INFO [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(4572): Replaying edits from hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/recovered.edits/0000000000000000093 2018-06-21 23:23:35,740 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(4760): Applied 379, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=93, path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/recovered.edits/0000000000000000093 2018-06-21 23:23:35,740 INFO [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(2560): Flushing 3/3 column families, dataSize=46.94 KB heapSize=74.55 KB 2018-06-21 23:23:35,799 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741884_1060{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|FINALIZED]]} size 0 2018-06-21 23:23:35,800 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741884_1060{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|FINALIZED]]} size 0 2018-06-21 23:23:35,800 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741884_1060{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|FINALIZED], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|FINALIZED]]} size 0 2018-06-21 23:23:35,801 INFO [RS_OPEN_META-master/asf911:0-0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=46.76 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/.tmp/info/76d2ae477a214062946362815649f5ab 2018-06-21 23:23:35,824 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741885_1061{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:35,824 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741885_1061{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|FINALIZED]]} size 0 2018-06-21 23:23:35,825 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741885_1061 size 4851 2018-06-21 23:23:35,825 INFO [RS_OPEN_META-master/asf911:0-0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=190 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/.tmp/table/cac52b561444446c972f1285f18c4aa4 2018-06-21 23:23:35,834 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/.tmp/info/76d2ae477a214062946362815649f5ab as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/info/76d2ae477a214062946362815649f5ab 2018-06-21 23:23:35,841 INFO [RS_OPEN_META-master/asf911:0-0] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/info/76d2ae477a214062946362815649f5ab, entries=375, sequenceid=93, filesize=52.4 K 2018-06-21 23:23:35,843 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/.tmp/table/cac52b561444446c972f1285f18c4aa4 as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/table/cac52b561444446c972f1285f18c4aa4 2018-06-21 23:23:35,851 INFO [RS_OPEN_META-master/asf911:0-0] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/table/cac52b561444446c972f1285f18c4aa4, entries=4, sequenceid=93, filesize=4.7 K 2018-06-21 23:23:35,851 INFO [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(2769): Finished flush of dataSize ~46.94 KB/48069, heapSize ~74.97 KB/76768, currentSize=0 B/0 for 1588230740 in 111ms, sequenceid=93, compaction requested=false; wal=null 2018-06-21 23:23:35,852 INFO [IPC Server handler 5 on 56139] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741882_1058 127.0.0.1:33402 127.0.0.1:54793 127.0.0.1:53553 2018-06-21 23:23:35,853 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(4552): Deleted recovered.edits file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/recovered.edits/0000000000000000093 2018-06-21 23:23:35,853 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(931): stopping wal replay for 1588230740 2018-06-21 23:23:35,853 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(943): Cleaning up temporary data for 1588230740 2018-06-21 23:23:35,854 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(954): Cleaning up detritus for 1588230740 2018-06-21 23:23:35,855 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7M)) instead. 2018-06-21 23:23:35,856 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(976): writing seq id for 1588230740 2018-06-21 23:23:35,860 DEBUG [RS_OPEN_META-master/asf911:0-0] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/recovered.edits/93.seqid, newMaxSeqId=93, maxSeqId=1 2018-06-21 23:23:35,860 INFO [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(980): Opened 1588230740; next sequenceid=94 2018-06-21 23:23:35,860 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(987): Running coprocessor post-open hooks for 1588230740 2018-06-21 23:23:35,862 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2187): Post open deploy tasks for hbase:meta,,1.1588230740 2018-06-21 23:23:35,866 DEBUG [PostOpenDeployTasks:1588230740] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=94, pid=56, ppid=55, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:35,866 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2211): Finished post open deploy task for hbase:meta,,1.1588230740 2018-06-21 23:23:35,866 DEBUG [PEWorker-13] assignment.RegionTransitionProcedure(354): Finishing pid=56, ppid=55, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:35,866 DEBUG [RS_OPEN_META-master/asf911:0-0] handler.OpenRegionHandler(128): Opened hbase:meta,,1.1588230740 on asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:35,867 INFO [PEWorker-13] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:36,048 INFO [PEWorker-13] procedure2.ProcedureExecutor(1635): Finished subprocedure(s) of pid=55, state=RUNNABLE:SERVER_CRASH_GET_REGIONS; ServerCrashProcedure server=asf911.gq1.ygridcore.net,40767,1529623390148, splitWal=true, meta=true; resume parent processing. 2018-06-21 23:23:36,048 INFO [PEWorker-13] procedure2.ProcedureExecutor(1266): Finished pid=56, ppid=55, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 in 590msec 2018-06-21 23:23:36,138 DEBUG [PEWorker-6] procedure.ServerCrashProcedure(239): Splitting WALs pid=55, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS; ServerCrashProcedure server=asf911.gq1.ygridcore.net,40767,1529623390148, splitWal=true, meta=true 2018-06-21 23:23:36,140 INFO [PEWorker-6] master.SplitLogManager(461): dead splitlog workers [asf911.gq1.ygridcore.net,40767,1529623390148] 2018-06-21 23:23:36,142 INFO [PEWorker-6] master.SplitLogManager(241): Started splitting 1 logs in [hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,40767,1529623390148-splitting] for [asf911.gq1.ygridcore.net,40767,1529623390148] 2018-06-21 23:23:36,150 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:36,150 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:36,150 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:36,150 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:36,150 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:36,150 DEBUG [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination(332): Put up splitlog task at znode /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.1529623393801 2018-06-21 23:23:36,151 DEBUG [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination(364): Task not yet acquired /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.1529623393801, ver=0 2018-06-21 23:23:36,259 INFO [master/asf911:0.splitLogManager..Chore.1] master.SplitLogManager$TimeoutMonitor(591): total=1, unassigned=1, tasks={/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.1529623393801=last_update = -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0} 2018-06-21 23:23:36,689 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-06-21 23:23:36,730 DEBUG [Default-IPC-NioEventLoopGroup-9-17] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:40767 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:40767 2018-06-21 23:23:36,936 DEBUG [SplitLogWorker-asf911:56613] coordination.ZkSplitLogWorkerCoordination(436): Current region server asf911.gq1.ygridcore.net,56613,1529623390297 is ready to take more tasks, will get task list and try grab tasks again. 2018-06-21 23:23:36,957 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.1529623393801 2018-06-21 23:23:36,957 INFO [SplitLogWorker-asf911:56613] coordination.ZkSplitLogWorkerCoordination(258): worker asf911.gq1.ygridcore.net,56613,1529623390297 acquired task /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.1529623393801 2018-06-21 23:23:36,957 INFO [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination(454): Task /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.1529623393801 acquired by asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:36,958 DEBUG [SplitLogWorker-asf911:52507] coordination.ZkSplitLogWorkerCoordination(436): Current region server asf911.gq1.ygridcore.net,52507,1529623390228 is ready to take more tasks, will get task list and try grab tasks again. 2018-06-21 23:23:36,958 DEBUG [SplitLogWorker-asf911:58235] coordination.ZkSplitLogWorkerCoordination(436): Current region server asf911.gq1.ygridcore.net,58235,1529623390382 is ready to take more tasks, will get task list and try grab tasks again. 2018-06-21 23:23:36,958 DEBUG [SplitLogWorker-asf911:34033] coordination.ZkSplitLogWorkerCoordination(436): Current region server asf911.gq1.ygridcore.net,34033,1529623407773 is ready to take more tasks, will get task list and try grab tasks again. 2018-06-21 23:23:36,976 DEBUG [hconnection-0xe1c5692-shared-pool15-t31] ipc.AbstractRpcClient(349): Not trying to connect to asf911.gq1.ygridcore.net/67.195.81.155:40767 this server is in the failed servers list 2018-06-21 23:23:36,981 INFO [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] wal.WALSplitter(250): Splitting WAL=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,40767,1529623390148-splitting/asf911.gq1.ygridcore.net%2C40767%2C1529623390148.1529623393801, length=17942 2018-06-21 23:23:36,991 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.1529623393801 2018-06-21 23:23:36,991 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.1529623393801 2018-06-21 23:23:36,992 INFO [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] util.FSHDFSUtils(195): Recover lease on dfs file hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,40767,1529623390148-splitting/asf911.gq1.ygridcore.net%2C40767%2C1529623390148.1529623393801 2018-06-21 23:23:36,992 DEBUG [SplitLogWorker-asf911:55146] coordination.ZkSplitLogWorkerCoordination(436): Current region server asf911.gq1.ygridcore.net,55146,1529623388534 is ready to take more tasks, will get task list and try grab tasks again. 2018-06-21 23:23:36,993 INFO [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] util.FSHDFSUtils(284): Recovered lease, attempt=0 on file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,40767,1529623390148-splitting/asf911.gq1.ygridcore.net%2C40767%2C1529623390148.1529623393801 after 1ms 2018-06-21 23:23:37,002 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] wal.WALSplitter(283): DLS Last flushed sequenceid for d4ac3b44d02d68507d6317e201672011: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "family" sequence_id: 3 } 2018-06-21 23:23:37,003 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] wal.WALSplitter(283): DLS Last flushed sequenceid for 36d0a953419e9f81636aebf73e7ed5df: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "family" sequence_id: 3 } 2018-06-21 23:23:37,004 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] wal.WALSplitter(283): DLS Last flushed sequenceid for c793c2c3948aad84865d30ebf71a529e: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "family" sequence_id: 3 } 2018-06-21 23:23:37,005 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] wal.WALSplitter(283): DLS Last flushed sequenceid for 91ab16f39c84b448bdc2e5a1e1fc801b: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "family" sequence_id: 3 } 2018-06-21 23:23:37,006 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] wal.WALSplitter(283): DLS Last flushed sequenceid for 40503e5e9a757a4ccf579bf29b8d78c6: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "family" sequence_id: 3 } 2018-06-21 23:23:37,011 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] wal.WALSplitter(283): DLS Last flushed sequenceid for 0dfb3ebb8c759d632ec793ac0387ec95: last_flushed_sequence_id: 5 store_sequence_id { family_name: "family" sequence_id: 7 } 2018-06-21 23:23:37,013 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] wal.WALSplitter(283): DLS Last flushed sequenceid for fee67ae5d4f2b2be35fa516877a8d173: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "family" sequence_id: 12 } 2018-06-21 23:23:37,014 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] wal.WALSplitter(335): Finishing writing output logs and closing down 2018-06-21 23:23:37,014 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] wal.WALSplitter$OutputSink(1162): Waiting for split writer threads to finish 2018-06-21 23:23:37,017 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0-Writer-2] wal.WALSplitter$LogRecoveredEditsOutputSink(1506): Creating writer path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/c793c2c3948aad84865d30ebf71a529e/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C40767%2C1529623390148.1529623393801.temp 2018-06-21 23:23:37,017 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0-Writer-0] wal.WALSplitter$LogRecoveredEditsOutputSink(1506): Creating writer path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/36d0a953419e9f81636aebf73e7ed5df/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C40767%2C1529623390148.1529623393801.temp 2018-06-21 23:23:37,020 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0-Writer-1] wal.WALSplitter$LogRecoveredEditsOutputSink(1506): Creating writer path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/91ab16f39c84b448bdc2e5a1e1fc801b/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C40767%2C1529623390148.1529623393801.temp 2018-06-21 23:23:37,032 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0-Writer-2] wal.WALSplitter$LogRecoveredEditsOutputSink(1506): Creating writer path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/d4ac3b44d02d68507d6317e201672011/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C40767%2C1529623390148.1529623393801.temp 2018-06-21 23:23:37,032 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0-Writer-1] wal.WALSplitter$LogRecoveredEditsOutputSink(1506): Creating writer path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/recovered.edits/0000000000000000008-asf911.gq1.ygridcore.net%2C40767%2C1529623390148.1529623393801.temp 2018-06-21 23:23:37,032 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0-Writer-0] wal.WALSplitter$LogRecoveredEditsOutputSink(1506): Creating writer path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/40503e5e9a757a4ccf579bf29b8d78c6/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C40767%2C1529623390148.1529623393801.temp 2018-06-21 23:23:37,039 DEBUG [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0-Writer-1] wal.WALSplitter$LogRecoveredEditsOutputSink(1506): Creating writer path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/recovered.edits/0000000000000000013-asf911.gq1.ygridcore.net%2C40767%2C1529623390148.1529623393801.temp 2018-06-21 23:23:37,039 INFO [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] wal.WALSplitter$OutputSink(1186): 3 split writers finished; closing. 2018-06-21 23:23:37,055 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741887_1063{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW]]} size 2755 2018-06-21 23:23:37,056 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741887_1063 size 2755 2018-06-21 23:23:37,056 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741888_1064{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW]]} size 2927 2018-06-21 23:23:37,056 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741886_1062{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW]]} size 2755 2018-06-21 23:23:37,056 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741886_1062 size 2755 2018-06-21 23:23:37,056 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741886_1062 size 2755 2018-06-21 23:23:37,057 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741887_1063 size 2755 2018-06-21 23:23:37,057 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741888_1064 size 2927 2018-06-21 23:23:37,057 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741888_1064 size 2927 2018-06-21 23:23:37,314 DEBUG [hconnection-0xe1c5692-shared-pool15-t31] ipc.AbstractRpcClient(349): Not trying to connect to asf911.gq1.ygridcore.net/67.195.81.155:40767 this server is in the failed servers list 2018-06-21 23:23:37,457 DEBUG [split-log-closeStream-1] wal.WALSplitter$LogRecoveredEditsOutputSink(1381): Closed wap hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/d4ac3b44d02d68507d6317e201672011/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C40767%2C1529623390148.1529623393801.temp (wrote 4 edits, skipped 0 edits in 12ms 2018-06-21 23:23:37,457 DEBUG [split-log-closeStream-2] wal.WALSplitter$LogRecoveredEditsOutputSink(1381): Closed wap hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/40503e5e9a757a4ccf579bf29b8d78c6/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C40767%2C1529623390148.1529623393801.temp (wrote 4 edits, skipped 0 edits in 17ms 2018-06-21 23:23:37,457 DEBUG [split-log-closeStream-3] wal.WALSplitter$LogRecoveredEditsOutputSink(1381): Closed wap hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/c793c2c3948aad84865d30ebf71a529e/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C40767%2C1529623390148.1529623393801.temp (wrote 4 edits, skipped 0 edits in 15ms 2018-06-21 23:23:37,460 INFO [split-log-closeStream-1] wal.WALSplitter$LogRecoveredEditsOutputSink(1407): Rename hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/d4ac3b44d02d68507d6317e201672011/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C40767%2C1529623390148.1529623393801.temp to hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/d4ac3b44d02d68507d6317e201672011/recovered.edits/0000000000000000007 2018-06-21 23:23:37,464 INFO [split-log-closeStream-3] wal.WALSplitter$LogRecoveredEditsOutputSink(1407): Rename hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/c793c2c3948aad84865d30ebf71a529e/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C40767%2C1529623390148.1529623393801.temp to hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/c793c2c3948aad84865d30ebf71a529e/recovered.edits/0000000000000000007 2018-06-21 23:23:37,464 INFO [split-log-closeStream-2] wal.WALSplitter$LogRecoveredEditsOutputSink(1407): Rename hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/40503e5e9a757a4ccf579bf29b8d78c6/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C40767%2C1529623390148.1529623393801.temp to hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/40503e5e9a757a4ccf579bf29b8d78c6/recovered.edits/0000000000000000007 2018-06-21 23:23:37,478 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741890_1066{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:37,478 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741890_1066{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:37,478 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741889_1065{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:37,479 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741889_1065{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:37,479 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741890_1066 size 1423 2018-06-21 23:23:37,479 DEBUG [split-log-closeStream-3] wal.WALSplitter$LogRecoveredEditsOutputSink(1381): Closed wap hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/recovered.edits/0000000000000000008-asf911.gq1.ygridcore.net%2C40767%2C1529623390148.1529623393801.temp (wrote 2 edits, skipped 0 edits in 6ms 2018-06-21 23:23:37,479 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741891_1067{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:37,480 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741889_1065{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:37,480 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741891_1067{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:37,480 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741891_1067{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:37,481 DEBUG [split-log-closeStream-1] wal.WALSplitter$LogRecoveredEditsOutputSink(1381): Closed wap hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/91ab16f39c84b448bdc2e5a1e1fc801b/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C40767%2C1529623390148.1529623393801.temp (wrote 4 edits, skipped 0 edits in 13ms 2018-06-21 23:23:37,481 DEBUG [split-log-closeStream-2] wal.WALSplitter$LogRecoveredEditsOutputSink(1381): Closed wap hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/36d0a953419e9f81636aebf73e7ed5df/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C40767%2C1529623390148.1529623393801.temp (wrote 4 edits, skipped 0 edits in 10ms 2018-06-21 23:23:37,483 INFO [split-log-closeStream-3] wal.WALSplitter$LogRecoveredEditsOutputSink(1407): Rename hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/recovered.edits/0000000000000000008-asf911.gq1.ygridcore.net%2C40767%2C1529623390148.1529623393801.temp to hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/recovered.edits/0000000000000000009 2018-06-21 23:23:37,519 INFO [split-log-closeStream-2] wal.WALSplitter$LogRecoveredEditsOutputSink(1407): Rename hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/36d0a953419e9f81636aebf73e7ed5df/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C40767%2C1529623390148.1529623393801.temp to hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/36d0a953419e9f81636aebf73e7ed5df/recovered.edits/0000000000000000007 2018-06-21 23:23:37,519 INFO [split-log-closeStream-1] wal.WALSplitter$LogRecoveredEditsOutputSink(1407): Rename hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/91ab16f39c84b448bdc2e5a1e1fc801b/recovered.edits/0000000000000000004-asf911.gq1.ygridcore.net%2C40767%2C1529623390148.1529623393801.temp to hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/91ab16f39c84b448bdc2e5a1e1fc801b/recovered.edits/0000000000000000007 2018-06-21 23:23:37,528 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741892_1068{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:37,528 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741892_1068{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:37,529 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741892_1068 size 757 2018-06-21 23:23:37,529 DEBUG [split-log-closeStream-3] wal.WALSplitter$LogRecoveredEditsOutputSink(1381): Closed wap hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/recovered.edits/0000000000000000013-asf911.gq1.ygridcore.net%2C40767%2C1529623390148.1529623393801.temp (wrote 1 edits, skipped 0 edits in 7ms 2018-06-21 23:23:37,532 INFO [split-log-closeStream-3] wal.WALSplitter$LogRecoveredEditsOutputSink(1407): Rename hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/recovered.edits/0000000000000000013-asf911.gq1.ygridcore.net%2C40767%2C1529623390148.1529623393801.temp to hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/recovered.edits/0000000000000000013 2018-06-21 23:23:37,532 INFO [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] wal.WALSplitter(356): Processed 23 edits across 7 regions; edits skipped=7; log file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,40767,1529623390148-splitting/asf911.gq1.ygridcore.net%2C40767%2C1529623390148.1529623393801, length=17942, corrupted=false, progress failed=false 2018-06-21 23:23:37,541 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.1529623393801 2018-06-21 23:23:37,541 INFO [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] coordination.ZkSplitLogWorkerCoordination(574): successfully transitioned task /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.1529623393801 to final state DONE asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:37,542 INFO [RS_LOG_REPLAY_OPS-regionserver/asf911:0-0] handler.WALSplitterHandler(99): Worker asf911.gq1.ygridcore.net,56613,1529623390297 done with task org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$ZkSplitTaskDetails@352b9776 in 585ms. Status = DONE 2018-06-21 23:23:37,541 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.1529623393801 2018-06-21 23:23:37,542 INFO [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination(372): Task /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.1529623393801 entered state=DONE asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:37,547 INFO [Time-limited test-EventThread] wal.WALSplitter(448): Archived processed log hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,40767,1529623390148-splitting/asf911.gq1.ygridcore.net%2C40767%2C1529623390148.1529623393801 to hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/oldWALs/asf911.gq1.ygridcore.net%2C40767%2C1529623390148.1529623393801 2018-06-21 23:23:37,548 INFO [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination(415): Done splitting /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.1529623393801 2018-06-21 23:23:37,550 INFO [PEWorker-6] master.SplitLogManager(293): finished splitting (more than or equal to) 17942 bytes in 1 log files in [hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,40767,1529623390148-splitting] in 1408ms 2018-06-21 23:23:37,550 DEBUG [PEWorker-6] procedure.ServerCrashProcedure(247): Done splitting WALs pid=55, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS; ServerCrashProcedure server=asf911.gq1.ygridcore.net,40767,1529623390148, splitWal=true, meta=true 2018-06-21 23:23:37,559 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.1529623393801 2018-06-21 23:23:37,559 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:37,559 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:37,559 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.1529623393801 2018-06-21 23:23:37,559 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:37,559 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:37,559 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-06-21 23:23:37,560 DEBUG [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination$DeleteAsyncCallback(696): Deleted /hbase/splitWAL/WALs%2Fasf911.gq1.ygridcore.net%2C40767%2C1529623390148-splitting%2Fasf911.gq1.ygridcore.net%252C40767%252C1529623390148.1529623393801 2018-06-21 23:23:37,622 INFO [PEWorker-6] procedure2.ProcedureExecutor(1516): Initialized subprocedures=[{pid=57, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=c793c2c3948aad84865d30ebf71a529e}, {pid=58, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=91ab16f39c84b448bdc2e5a1e1fc801b}, {pid=59, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=fee67ae5d4f2b2be35fa516877a8d173}, {pid=60, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=36d0a953419e9f81636aebf73e7ed5df}, {pid=61, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=d4ac3b44d02d68507d6317e201672011}, {pid=62, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=40503e5e9a757a4ccf579bf29b8d78c6}, {pid=63, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=0dfb3ebb8c759d632ec793ac0387ec95}] 2018-06-21 23:23:37,705 INFO [PEWorker-14] procedure.MasterProcedureScheduler(696): pid=62, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=40503e5e9a757a4ccf579bf29b8d78c6 checking lock on 40503e5e9a757a4ccf579bf29b8d78c6 2018-06-21 23:23:37,705 INFO [PEWorker-15] procedure.MasterProcedureScheduler(696): pid=61, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=d4ac3b44d02d68507d6317e201672011 checking lock on d4ac3b44d02d68507d6317e201672011 2018-06-21 23:23:37,705 INFO [PEWorker-8] procedure.MasterProcedureScheduler(696): pid=60, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=36d0a953419e9f81636aebf73e7ed5df checking lock on 36d0a953419e9f81636aebf73e7ed5df 2018-06-21 23:23:37,706 INFO [PEWorker-10] procedure.MasterProcedureScheduler(696): pid=57, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=c793c2c3948aad84865d30ebf71a529e checking lock on c793c2c3948aad84865d30ebf71a529e 2018-06-21 23:23:37,706 INFO [PEWorker-3] procedure.MasterProcedureScheduler(696): pid=63, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=0dfb3ebb8c759d632ec793ac0387ec95 checking lock on 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:37,706 INFO [PEWorker-7] procedure.MasterProcedureScheduler(696): pid=59, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=fee67ae5d4f2b2be35fa516877a8d173 checking lock on fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:37,709 INFO [PEWorker-16] procedure.MasterProcedureScheduler(696): pid=58, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=91ab16f39c84b448bdc2e5a1e1fc801b checking lock on 91ab16f39c84b448bdc2e5a1e1fc801b 2018-06-21 23:23:37,712 DEBUG [RS-EventLoopGroup-7-17] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:40767 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:40767 2018-06-21 23:23:37,717 DEBUG [RS-EventLoopGroup-7-26] ipc.FailedServers(56): Added failed server with address asf911.gq1.ygridcore.net/67.195.81.155:40767 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: asf911.gq1.ygridcore.net/67.195.81.155:40767 2018-06-21 23:23:37,830 INFO [PEWorker-16] assignment.AssignProcedure(218): Starting pid=58, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=91ab16f39c84b448bdc2e5a1e1fc801b; rit=OFFLINE, location=asf911.gq1.ygridcore.net,40767,1529623390148; forceNewPlan=false, retain=true 2018-06-21 23:23:37,830 INFO [PEWorker-14] assignment.AssignProcedure(218): Starting pid=62, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=40503e5e9a757a4ccf579bf29b8d78c6; rit=OFFLINE, location=asf911.gq1.ygridcore.net,40767,1529623390148; forceNewPlan=false, retain=true 2018-06-21 23:23:37,830 INFO [PEWorker-15] assignment.AssignProcedure(218): Starting pid=61, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=d4ac3b44d02d68507d6317e201672011; rit=OFFLINE, location=asf911.gq1.ygridcore.net,40767,1529623390148; forceNewPlan=false, retain=true 2018-06-21 23:23:37,830 INFO [PEWorker-3] assignment.AssignProcedure(218): Starting pid=63, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=0dfb3ebb8c759d632ec793ac0387ec95; rit=OFFLINE, location=asf911.gq1.ygridcore.net,40767,1529623390148; forceNewPlan=false, retain=true 2018-06-21 23:23:37,830 INFO [PEWorker-8] assignment.AssignProcedure(218): Starting pid=60, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=36d0a953419e9f81636aebf73e7ed5df; rit=OFFLINE, location=asf911.gq1.ygridcore.net,40767,1529623390148; forceNewPlan=false, retain=true 2018-06-21 23:23:37,830 INFO [PEWorker-10] assignment.AssignProcedure(218): Starting pid=57, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=c793c2c3948aad84865d30ebf71a529e; rit=OFFLINE, location=asf911.gq1.ygridcore.net,40767,1529623390148; forceNewPlan=false, retain=true 2018-06-21 23:23:37,830 INFO [PEWorker-7] assignment.AssignProcedure(218): Starting pid=59, ppid=55, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=multi_test_table, region=fee67ae5d4f2b2be35fa516877a8d173; rit=OFFLINE, location=asf911.gq1.ygridcore.net,40767,1529623390148; forceNewPlan=false, retain=true 2018-06-21 23:23:37,850 DEBUG [hconnection-0xe1c5692-shared-pool15-t31] ipc.AbstractRpcClient(349): Not trying to connect to asf911.gq1.ygridcore.net/67.195.81.155:40767 this server is in the failed servers list 2018-06-21 23:23:37,966 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@69ed135] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:33402 to delete [blk_1073741882_1058] 2018-06-21 23:23:37,966 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@69ed135] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:54793 to delete [blk_1073741882_1058] 2018-06-21 23:23:37,982 INFO [master/asf911:0] balancer.BaseLoadBalancer(1497): Reassigned 7 regions. 0 retained the pre-restart assignment. 7 regions were assigned to random hosts, since the old hosts for these regions are no longer present in the cluster. These hosts were: 2018-06-21 23:23:37,983 INFO [PEWorker-11] assignment.RegionStateStore(198): pid=59 updating hbase:meta row=fee67ae5d4f2b2be35fa516877a8d173, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:37,983 INFO [PEWorker-12] assignment.RegionStateStore(198): pid=57 updating hbase:meta row=c793c2c3948aad84865d30ebf71a529e, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:37,983 INFO [PEWorker-13] assignment.RegionStateStore(198): pid=61 updating hbase:meta row=d4ac3b44d02d68507d6317e201672011, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:37,983 INFO [PEWorker-4] assignment.RegionStateStore(198): pid=62 updating hbase:meta row=40503e5e9a757a4ccf579bf29b8d78c6, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:37,983 INFO [PEWorker-9] assignment.RegionStateStore(198): pid=63 updating hbase:meta row=0dfb3ebb8c759d632ec793ac0387ec95, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:37,984 INFO [PEWorker-5] assignment.RegionStateStore(198): pid=58 updating hbase:meta row=91ab16f39c84b448bdc2e5a1e1fc801b, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:37,984 INFO [PEWorker-1] assignment.RegionStateStore(198): pid=60 updating hbase:meta row=36d0a953419e9f81636aebf73e7ed5df, regionState=OPENING, regionLocation=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:37,987 INFO [PEWorker-12] assignment.RegionTransitionProcedure(241): Dispatch pid=57, ppid=55, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=c793c2c3948aad84865d30ebf71a529e; rit=OPENING, location=asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:37,987 INFO [PEWorker-11] assignment.RegionTransitionProcedure(241): Dispatch pid=59, ppid=55, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=fee67ae5d4f2b2be35fa516877a8d173; rit=OPENING, location=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:37,987 INFO [PEWorker-1] assignment.RegionTransitionProcedure(241): Dispatch pid=60, ppid=55, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=36d0a953419e9f81636aebf73e7ed5df; rit=OPENING, location=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:37,987 INFO [PEWorker-5] assignment.RegionTransitionProcedure(241): Dispatch pid=58, ppid=55, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=91ab16f39c84b448bdc2e5a1e1fc801b; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:37,987 INFO [PEWorker-9] assignment.RegionTransitionProcedure(241): Dispatch pid=63, ppid=55, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=0dfb3ebb8c759d632ec793ac0387ec95; rit=OPENING, location=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:37,987 INFO [PEWorker-4] assignment.RegionTransitionProcedure(241): Dispatch pid=62, ppid=55, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=40503e5e9a757a4ccf579bf29b8d78c6; rit=OPENING, location=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:37,987 INFO [PEWorker-13] assignment.RegionTransitionProcedure(241): Dispatch pid=61, ppid=55, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=d4ac3b44d02d68507d6317e201672011; rit=OPENING, location=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:38,140 DEBUG [RSProcedureDispatcher-pool3-t32] master.ServerManager(746): New admin connection to asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:38,143 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=58235] regionserver.RSRpcServices(1980): Open multi_test_table,hhh,1529623399386.91ab16f39c84b448bdc2e5a1e1fc801b. 2018-06-21 23:23:38,143 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=52507] regionserver.RSRpcServices(1980): Open multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173. 2018-06-21 23:23:38,143 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=56613] regionserver.RSRpcServices(1980): Open multi_test_table,ppp,1529623399386.36d0a953419e9f81636aebf73e7ed5df. 2018-06-21 23:23:38,144 INFO [RS-EventLoopGroup-10-15] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-06-21 23:23:38,145 INFO [RS-EventLoopGroup-10-15] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 52316 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:38,146 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=34033] regionserver.RSRpcServices(1980): Open multi_test_table,eee,1529623399386.c793c2c3948aad84865d30ebf71a529e. 2018-06-21 23:23:38,147 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=56613] regionserver.RSRpcServices(1980): Open multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95. 2018-06-21 23:23:38,147 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=52507] regionserver.RSRpcServices(1980): Open multi_test_table,sss,1529623399386.d4ac3b44d02d68507d6317e201672011. 2018-06-21 23:23:38,148 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7094): Opening region: {ENCODED => 91ab16f39c84b448bdc2e5a1e1fc801b, NAME => 'multi_test_table,hhh,1529623399386.91ab16f39c84b448bdc2e5a1e1fc801b.', STARTKEY => 'hhh', ENDKEY => 'iii'} 2018-06-21 23:23:38,148 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7094): Opening region: {ENCODED => 0dfb3ebb8c759d632ec793ac0387ec95, NAME => 'multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95.', STARTKEY => 'vvv', ENDKEY => 'www'} 2018-06-21 23:23:38,148 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7094): Opening region: {ENCODED => d4ac3b44d02d68507d6317e201672011, NAME => 'multi_test_table,sss,1529623399386.d4ac3b44d02d68507d6317e201672011.', STARTKEY => 'sss', ENDKEY => 'ttt'} 2018-06-21 23:23:38,148 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7094): Opening region: {ENCODED => 36d0a953419e9f81636aebf73e7ed5df, NAME => 'multi_test_table,ppp,1529623399386.36d0a953419e9f81636aebf73e7ed5df.', STARTKEY => 'ppp', ENDKEY => 'qqq'} 2018-06-21 23:23:38,148 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table 91ab16f39c84b448bdc2e5a1e1fc801b 2018-06-21 23:23:38,149 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:38,148 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7094): Opening region: {ENCODED => fee67ae5d4f2b2be35fa516877a8d173, NAME => 'multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173.', STARTKEY => 'kkk', ENDKEY => 'lll'} 2018-06-21 23:23:38,148 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=52507] regionserver.RSRpcServices(1980): Open multi_test_table,uuu,1529623399386.40503e5e9a757a4ccf579bf29b8d78c6. 2018-06-21 23:23:38,149 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(827): Instantiated multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:38,149 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(827): Instantiated multi_test_table,hhh,1529623399386.91ab16f39c84b448bdc2e5a1e1fc801b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:38,149 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table 36d0a953419e9f81636aebf73e7ed5df 2018-06-21 23:23:38,149 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table d4ac3b44d02d68507d6317e201672011 2018-06-21 23:23:38,153 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(827): Instantiated multi_test_table,ppp,1529623399386.36d0a953419e9f81636aebf73e7ed5df.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:38,152 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7134): checking encryption for 91ab16f39c84b448bdc2e5a1e1fc801b 2018-06-21 23:23:38,153 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7139): checking classloading for 91ab16f39c84b448bdc2e5a1e1fc801b 2018-06-21 23:23:38,152 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7094): Opening region: {ENCODED => 40503e5e9a757a4ccf579bf29b8d78c6, NAME => 'multi_test_table,uuu,1529623399386.40503e5e9a757a4ccf579bf29b8d78c6.', STARTKEY => 'uuu', ENDKEY => 'vvv'} 2018-06-21 23:23:38,152 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7134): checking encryption for 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:38,153 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7134): checking encryption for 36d0a953419e9f81636aebf73e7ed5df 2018-06-21 23:23:38,153 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(827): Instantiated multi_test_table,sss,1529623399386.d4ac3b44d02d68507d6317e201672011.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:38,153 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table 40503e5e9a757a4ccf579bf29b8d78c6 2018-06-21 23:23:38,154 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(827): Instantiated multi_test_table,uuu,1529623399386.40503e5e9a757a4ccf579bf29b8d78c6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:38,154 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7134): checking encryption for 40503e5e9a757a4ccf579bf29b8d78c6 2018-06-21 23:23:38,154 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(7139): checking classloading for 40503e5e9a757a4ccf579bf29b8d78c6 2018-06-21 23:23:38,153 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7139): checking classloading for 36d0a953419e9f81636aebf73e7ed5df 2018-06-21 23:23:38,153 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7139): checking classloading for 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:38,153 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7134): checking encryption for d4ac3b44d02d68507d6317e201672011 2018-06-21 23:23:38,155 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(7139): checking classloading for d4ac3b44d02d68507d6317e201672011 2018-06-21 23:23:38,155 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7094): Opening region: {ENCODED => c793c2c3948aad84865d30ebf71a529e, NAME => 'multi_test_table,eee,1529623399386.c793c2c3948aad84865d30ebf71a529e.', STARTKEY => 'eee', ENDKEY => 'fff'} 2018-06-21 23:23:38,156 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table c793c2c3948aad84865d30ebf71a529e 2018-06-21 23:23:38,156 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(827): Instantiated multi_test_table,eee,1529623399386.c793c2c3948aad84865d30ebf71a529e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:38,156 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table multi_test_table fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:38,156 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(827): Instantiated multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:38,157 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7134): checking encryption for fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:38,157 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7139): checking classloading for fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:38,156 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7134): checking encryption for c793c2c3948aad84865d30ebf71a529e 2018-06-21 23:23:38,158 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(7139): checking classloading for c793c2c3948aad84865d30ebf71a529e 2018-06-21 23:23:38,158 DEBUG [StoreOpener-36d0a953419e9f81636aebf73e7ed5df-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/36d0a953419e9f81636aebf73e7ed5df/family 2018-06-21 23:23:38,158 DEBUG [StoreOpener-36d0a953419e9f81636aebf73e7ed5df-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/36d0a953419e9f81636aebf73e7ed5df/family 2018-06-21 23:23:38,158 DEBUG [StoreOpener-40503e5e9a757a4ccf579bf29b8d78c6-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/40503e5e9a757a4ccf579bf29b8d78c6/family 2018-06-21 23:23:38,158 DEBUG [StoreOpener-40503e5e9a757a4ccf579bf29b8d78c6-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/40503e5e9a757a4ccf579bf29b8d78c6/family 2018-06-21 23:23:38,158 DEBUG [StoreOpener-91ab16f39c84b448bdc2e5a1e1fc801b-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/91ab16f39c84b448bdc2e5a1e1fc801b/family 2018-06-21 23:23:38,158 DEBUG [StoreOpener-91ab16f39c84b448bdc2e5a1e1fc801b-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/91ab16f39c84b448bdc2e5a1e1fc801b/family 2018-06-21 23:23:38,159 INFO [StoreOpener-36d0a953419e9f81636aebf73e7ed5df-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=11, currentSize=794.93 KB, freeSize=994.82 MB, maxSize=995.60 MB, heapSize=794.93 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:38,159 DEBUG [StoreOpener-d4ac3b44d02d68507d6317e201672011-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/d4ac3b44d02d68507d6317e201672011/family 2018-06-21 23:23:38,159 DEBUG [StoreOpener-d4ac3b44d02d68507d6317e201672011-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/d4ac3b44d02d68507d6317e201672011/family 2018-06-21 23:23:38,159 INFO [StoreOpener-91ab16f39c84b448bdc2e5a1e1fc801b-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=11, currentSize=794.93 KB, freeSize=994.82 MB, maxSize=995.60 MB, heapSize=794.93 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:38,159 INFO [StoreOpener-36d0a953419e9f81636aebf73e7ed5df-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:38,159 DEBUG [StoreOpener-0dfb3ebb8c759d632ec793ac0387ec95-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/family 2018-06-21 23:23:38,160 DEBUG [StoreOpener-0dfb3ebb8c759d632ec793ac0387ec95-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/family 2018-06-21 23:23:38,160 INFO [StoreOpener-d4ac3b44d02d68507d6317e201672011-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=11, currentSize=794.93 KB, freeSize=994.82 MB, maxSize=995.60 MB, heapSize=794.93 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:38,160 INFO [StoreOpener-91ab16f39c84b448bdc2e5a1e1fc801b-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:38,160 INFO [StoreOpener-40503e5e9a757a4ccf579bf29b8d78c6-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=11, currentSize=794.93 KB, freeSize=994.82 MB, maxSize=995.60 MB, heapSize=794.93 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:38,160 INFO [StoreOpener-0dfb3ebb8c759d632ec793ac0387ec95-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=11, currentSize=794.93 KB, freeSize=994.82 MB, maxSize=995.60 MB, heapSize=794.93 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:38,160 INFO [StoreOpener-36d0a953419e9f81636aebf73e7ed5df-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:38,161 INFO [StoreOpener-d4ac3b44d02d68507d6317e201672011-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:38,161 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(923): replaying wal for 36d0a953419e9f81636aebf73e7ed5df 2018-06-21 23:23:38,161 INFO [StoreOpener-40503e5e9a757a4ccf579bf29b8d78c6-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:38,161 INFO [StoreOpener-91ab16f39c84b448bdc2e5a1e1fc801b-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:38,161 INFO [StoreOpener-0dfb3ebb8c759d632ec793ac0387ec95-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:38,161 DEBUG [StoreOpener-fee67ae5d4f2b2be35fa516877a8d173-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/family 2018-06-21 23:23:38,162 DEBUG [StoreOpener-fee67ae5d4f2b2be35fa516877a8d173-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/family 2018-06-21 23:23:38,162 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(923): replaying wal for 91ab16f39c84b448bdc2e5a1e1fc801b 2018-06-21 23:23:38,162 INFO [StoreOpener-d4ac3b44d02d68507d6317e201672011-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:38,162 INFO [StoreOpener-fee67ae5d4f2b2be35fa516877a8d173-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=11, currentSize=794.93 KB, freeSize=994.82 MB, maxSize=995.60 MB, heapSize=794.93 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:38,163 INFO [StoreOpener-40503e5e9a757a4ccf579bf29b8d78c6-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:38,163 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(923): replaying wal for d4ac3b44d02d68507d6317e201672011 2018-06-21 23:23:38,163 INFO [StoreOpener-fee67ae5d4f2b2be35fa516877a8d173-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:38,163 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(923): replaying wal for 40503e5e9a757a4ccf579bf29b8d78c6 2018-06-21 23:23:38,164 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(4475): Found 1 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/36d0a953419e9f81636aebf73e7ed5df 2018-06-21 23:23:38,165 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(4475): Found 1 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/91ab16f39c84b448bdc2e5a1e1fc801b 2018-06-21 23:23:38,166 DEBUG [StoreOpener-c793c2c3948aad84865d30ebf71a529e-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/c793c2c3948aad84865d30ebf71a529e/family 2018-06-21 23:23:38,166 DEBUG [StoreOpener-c793c2c3948aad84865d30ebf71a529e-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/c793c2c3948aad84865d30ebf71a529e/family 2018-06-21 23:23:38,166 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(4572): Replaying edits from hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/36d0a953419e9f81636aebf73e7ed5df/recovered.edits/0000000000000000007 2018-06-21 23:23:38,166 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(4475): Found 1 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/40503e5e9a757a4ccf579bf29b8d78c6 2018-06-21 23:23:38,166 INFO [StoreOpener-c793c2c3948aad84865d30ebf71a529e-1] hfile.CacheConfig(239): Created cacheConfig for family: blockCache=LruBlockCache{blockCount=11, currentSize=794.93 KB, freeSize=994.82 MB, maxSize=995.60 MB, heapSize=794.93 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:38,167 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(4475): Found 1 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/d4ac3b44d02d68507d6317e201672011 2018-06-21 23:23:38,167 INFO [StoreOpener-c793c2c3948aad84865d30ebf71a529e-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:38,167 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(4572): Replaying edits from hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/91ab16f39c84b448bdc2e5a1e1fc801b/recovered.edits/0000000000000000007 2018-06-21 23:23:38,168 INFO [StoreOpener-c793c2c3948aad84865d30ebf71a529e-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:38,168 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(4572): Replaying edits from hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/d4ac3b44d02d68507d6317e201672011/recovered.edits/0000000000000000007 2018-06-21 23:23:38,168 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(923): replaying wal for c793c2c3948aad84865d30ebf71a529e 2018-06-21 23:23:38,168 INFO [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(4572): Replaying edits from hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/40503e5e9a757a4ccf579bf29b8d78c6/recovered.edits/0000000000000000007 2018-06-21 23:23:38,174 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(4760): Applied 56, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=7, path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/36d0a953419e9f81636aebf73e7ed5df/recovered.edits/0000000000000000007 2018-06-21 23:23:38,174 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=2.13 KB heapSize=4.38 KB 2018-06-21 23:23:38,175 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(4475): Found 1 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/c793c2c3948aad84865d30ebf71a529e 2018-06-21 23:23:38,175 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(4760): Applied 60, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=7, path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/91ab16f39c84b448bdc2e5a1e1fc801b/recovered.edits/0000000000000000007 2018-06-21 23:23:38,176 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=2.29 KB heapSize=4.81 KB 2018-06-21 23:23:38,176 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(4572): Replaying edits from hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/c793c2c3948aad84865d30ebf71a529e/recovered.edits/0000000000000000007 2018-06-21 23:23:38,179 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(4760): Applied 56, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=7, path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/40503e5e9a757a4ccf579bf29b8d78c6/recovered.edits/0000000000000000007 2018-06-21 23:23:38,179 INFO [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=2.13 KB heapSize=4.38 KB 2018-06-21 23:23:38,179 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(4760): Applied 56, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=7, path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/d4ac3b44d02d68507d6317e201672011/recovered.edits/0000000000000000007 2018-06-21 23:23:38,180 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=2.13 KB heapSize=4.38 KB 2018-06-21 23:23:38,180 DEBUG [StoreOpener-fee67ae5d4f2b2be35fa516877a8d173-1] regionserver.HStore(581): loaded hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/family/7faefec2067642c89a50ae799bba98ba 2018-06-21 23:23:38,180 INFO [StoreOpener-fee67ae5d4f2b2be35fa516877a8d173-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:38,180 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(923): replaying wal for fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:38,184 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(4475): Found 1 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:38,184 DEBUG [StoreOpener-0dfb3ebb8c759d632ec793ac0387ec95-1] regionserver.HStore(581): loaded hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/family/a9b2590c1fd74168ad2e74d2f3c0070d 2018-06-21 23:23:38,184 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(4760): Applied 60, skipped 0, firstSequenceIdInLog=4, maxSequenceIdInLog=7, path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/c793c2c3948aad84865d30ebf71a529e/recovered.edits/0000000000000000007 2018-06-21 23:23:38,184 INFO [StoreOpener-0dfb3ebb8c759d632ec793ac0387ec95-1] regionserver.HStore(327): Store=family, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:38,185 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=2.29 KB heapSize=4.81 KB 2018-06-21 23:23:38,185 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(923): replaying wal for 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:38,185 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(4572): Replaying edits from hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/recovered.edits/0000000000000000013 2018-06-21 23:23:38,187 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(4475): Found 1 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:38,189 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(4572): Replaying edits from hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/recovered.edits/0000000000000000009 2018-06-21 23:23:38,189 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(4760): Applied 14, skipped 0, firstSequenceIdInLog=13, maxSequenceIdInLog=13, path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/recovered.edits/0000000000000000013 2018-06-21 23:23:38,190 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=546 B heapSize=1.09 KB 2018-06-21 23:23:38,196 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(4760): Applied 28, skipped 0, firstSequenceIdInLog=8, maxSequenceIdInLog=9, path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/recovered.edits/0000000000000000009 2018-06-21 23:23:38,197 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=1.07 KB heapSize=2.19 KB 2018-06-21 23:23:38,208 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741893_1069{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW]]} size 6592 2018-06-21 23:23:38,209 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741893_1069 size 6592 2018-06-21 23:23:38,209 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741893_1069 size 6592 2018-06-21 23:23:38,224 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741896_1072{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW]]} size 0 2018-06-21 23:23:38,224 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741896_1072{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW]]} size 0 2018-06-21 23:23:38,225 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741896_1072 size 6432 2018-06-21 23:23:38,226 INFO [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=2.13 KB at sequenceid=7 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/40503e5e9a757a4ccf579bf29b8d78c6/.tmp/family/f02539f7366f48d599254052b8edcae6 2018-06-21 23:23:38,230 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741898_1074{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:38,230 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741898_1074{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:38,232 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741898_1074 size 5632 2018-06-21 23:23:38,235 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741894_1070{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW]]} size 0 2018-06-21 23:23:38,236 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741894_1070 size 6592 2018-06-21 23:23:38,236 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741894_1070 size 6592 2018-06-21 23:23:38,239 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=2.29 KB at sequenceid=7 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/c793c2c3948aad84865d30ebf71a529e/.tmp/family/35d17e1206924343b27560ae79642b8f 2018-06-21 23:23:38,240 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=1.07 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/.tmp/family/0d2af5f77db842b3b0410b833fff1e83 2018-06-21 23:23:38,242 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741895_1071{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|FINALIZED]]} size 0 2018-06-21 23:23:38,243 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741895_1071{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|FINALIZED]]} size 0 2018-06-21 23:23:38,243 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/40503e5e9a757a4ccf579bf29b8d78c6/.tmp/family/f02539f7366f48d599254052b8edcae6 as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/40503e5e9a757a4ccf579bf29b8d78c6/family/f02539f7366f48d599254052b8edcae6 2018-06-21 23:23:38,243 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741895_1071{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|FINALIZED]]} size 0 2018-06-21 23:23:38,244 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741899_1075{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW]]} size 0 2018-06-21 23:23:38,245 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=2.13 KB at sequenceid=7 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/d4ac3b44d02d68507d6317e201672011/.tmp/family/505db391b1184e37a9da64c4286f9b35 2018-06-21 23:23:38,246 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=546 B at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/.tmp/family/4e8d6435f7d6452799530f9339fffaad 2018-06-21 23:23:38,246 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741899_1075 size 5232 2018-06-21 23:23:38,246 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741899_1075 size 5232 2018-06-21 23:23:38,250 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741897_1073{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW]]} size 0 2018-06-21 23:23:38,251 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741897_1073 size 6432 2018-06-21 23:23:38,251 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741897_1073 size 6432 2018-06-21 23:23:38,251 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=2.13 KB at sequenceid=7 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/36d0a953419e9f81636aebf73e7ed5df/.tmp/family/345c90f47b4d42e98700fa2e17b7f4fb 2018-06-21 23:23:38,260 INFO [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/40503e5e9a757a4ccf579bf29b8d78c6/family/f02539f7366f48d599254052b8edcae6, entries=40, sequenceid=7, filesize=6.3 K 2018-06-21 23:23:38,260 INFO [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(2769): Finished flush of dataSize ~2.13 KB/2184, heapSize ~4.59 KB/4696, currentSize=0 B/0 for 40503e5e9a757a4ccf579bf29b8d78c6 in 81ms, sequenceid=7, compaction requested=false; wal=null 2018-06-21 23:23:38,262 INFO [IPC Server handler 8 on 56139] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741887_1063 127.0.0.1:57095 127.0.0.1:33402 127.0.0.1:42074 2018-06-21 23:23:38,263 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/.tmp/family/0d2af5f77db842b3b0410b833fff1e83 as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/family/0d2af5f77db842b3b0410b833fff1e83 2018-06-21 23:23:38,263 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/c793c2c3948aad84865d30ebf71a529e/.tmp/family/35d17e1206924343b27560ae79642b8f as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/c793c2c3948aad84865d30ebf71a529e/family/35d17e1206924343b27560ae79642b8f 2018-06-21 23:23:38,263 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(4552): Deleted recovered.edits file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/40503e5e9a757a4ccf579bf29b8d78c6/recovered.edits/0000000000000000007 2018-06-21 23:23:38,264 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(931): stopping wal replay for 40503e5e9a757a4ccf579bf29b8d78c6 2018-06-21 23:23:38,265 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(943): Cleaning up temporary data for 40503e5e9a757a4ccf579bf29b8d78c6 2018-06-21 23:23:38,266 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(954): Cleaning up detritus for 40503e5e9a757a4ccf579bf29b8d78c6 2018-06-21 23:23:38,267 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/36d0a953419e9f81636aebf73e7ed5df/.tmp/family/345c90f47b4d42e98700fa2e17b7f4fb as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/36d0a953419e9f81636aebf73e7ed5df/family/345c90f47b4d42e98700fa2e17b7f4fb 2018-06-21 23:23:38,268 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/d4ac3b44d02d68507d6317e201672011/.tmp/family/505db391b1184e37a9da64c4286f9b35 as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/d4ac3b44d02d68507d6317e201672011/family/505db391b1184e37a9da64c4286f9b35 2018-06-21 23:23:38,268 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/.tmp/family/4e8d6435f7d6452799530f9339fffaad as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/family/4e8d6435f7d6452799530f9339fffaad 2018-06-21 23:23:38,270 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(976): writing seq id for 40503e5e9a757a4ccf579bf29b8d78c6 2018-06-21 23:23:38,276 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/40503e5e9a757a4ccf579bf29b8d78c6/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=1 2018-06-21 23:23:38,278 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/c793c2c3948aad84865d30ebf71a529e/family/35d17e1206924343b27560ae79642b8f, entries=44, sequenceid=7, filesize=6.4 K 2018-06-21 23:23:38,279 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(2769): Finished flush of dataSize ~2.29 KB/2340, heapSize ~5.02 KB/5144, currentSize=0 B/0 for c793c2c3948aad84865d30ebf71a529e in 95ms, sequenceid=7, compaction requested=false; wal=null 2018-06-21 23:23:38,280 INFO [IPC Server handler 5 on 56139] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741888_1064 127.0.0.1:42074 127.0.0.1:54793 127.0.0.1:33402 2018-06-21 23:23:38,280 INFO [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(980): Opened 40503e5e9a757a4ccf579bf29b8d78c6; next sequenceid=8 2018-06-21 23:23:38,280 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] regionserver.HRegion(987): Running coprocessor post-open hooks for 40503e5e9a757a4ccf579bf29b8d78c6 2018-06-21 23:23:38,280 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(4552): Deleted recovered.edits file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/c793c2c3948aad84865d30ebf71a529e/recovered.edits/0000000000000000007 2018-06-21 23:23:38,280 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(931): stopping wal replay for c793c2c3948aad84865d30ebf71a529e 2018-06-21 23:23:38,280 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(943): Cleaning up temporary data for c793c2c3948aad84865d30ebf71a529e 2018-06-21 23:23:38,281 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/family/4e8d6435f7d6452799530f9339fffaad, entries=10, sequenceid=13, filesize=5.1 K 2018-06-21 23:23:38,282 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(2769): Finished flush of dataSize ~546 B/546, heapSize ~1.30 KB/1336, currentSize=0 B/0 for fee67ae5d4f2b2be35fa516877a8d173 in 92ms, sequenceid=13, compaction requested=false; wal=null 2018-06-21 23:23:38,282 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/family/0d2af5f77db842b3b0410b833fff1e83, entries=20, sequenceid=9, filesize=5.5 K 2018-06-21 23:23:38,282 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(2769): Finished flush of dataSize ~1.07 KB/1092, heapSize ~2.40 KB/2456, currentSize=0 B/0 for 0dfb3ebb8c759d632ec793ac0387ec95 in 86ms, sequenceid=9, compaction requested=false; wal=null 2018-06-21 23:23:38,282 INFO [PostOpenDeployTasks:40503e5e9a757a4ccf579bf29b8d78c6] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,uuu,1529623399386.40503e5e9a757a4ccf579bf29b8d78c6. 2018-06-21 23:23:38,291 INFO [IPC Server handler 9 on 56139] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741892_1068 127.0.0.1:42074 127.0.0.1:33402 127.0.0.1:57095 2018-06-21 23:23:38,292 INFO [IPC Server handler 1 on 56139] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741890_1066 127.0.0.1:54793 127.0.0.1:53553 127.0.0.1:57095 2018-06-21 23:23:38,292 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(4552): Deleted recovered.edits file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/recovered.edits/0000000000000000013 2018-06-21 23:23:38,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=8, pid=62, ppid=55, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=40503e5e9a757a4ccf579bf29b8d78c6; rit=OPENING, location=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:38,292 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(954): Cleaning up detritus for c793c2c3948aad84865d30ebf71a529e 2018-06-21 23:23:38,292 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(931): stopping wal replay for fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:38,292 DEBUG [PEWorker-2] assignment.RegionTransitionProcedure(354): Finishing pid=62, ppid=55, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=40503e5e9a757a4ccf579bf29b8d78c6; rit=OPENING, location=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:38,293 DEBUG [PostOpenDeployTasks:40503e5e9a757a4ccf579bf29b8d78c6] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,uuu,1529623399386.40503e5e9a757a4ccf579bf29b8d78c6. 2018-06-21 23:23:38,293 INFO [PEWorker-2] assignment.RegionStateStore(198): pid=62 updating hbase:meta row=40503e5e9a757a4ccf579bf29b8d78c6, regionState=OPEN, openSeqNum=8, regionLocation=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:38,292 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/36d0a953419e9f81636aebf73e7ed5df/family/345c90f47b4d42e98700fa2e17b7f4fb, entries=40, sequenceid=7, filesize=6.3 K 2018-06-21 23:23:38,293 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-2] handler.OpenRegionHandler(128): Opened multi_test_table,uuu,1529623399386.40503e5e9a757a4ccf579bf29b8d78c6. on asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:38,293 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(943): Cleaning up temporary data for fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:38,294 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(2769): Finished flush of dataSize ~2.13 KB/2184, heapSize ~4.59 KB/4696, currentSize=0 B/0 for 36d0a953419e9f81636aebf73e7ed5df in 120ms, sequenceid=7, compaction requested=false; wal=null 2018-06-21 23:23:38,295 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(4552): Deleted recovered.edits file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/recovered.edits/0000000000000000009 2018-06-21 23:23:38,296 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(954): Cleaning up detritus for fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:38,296 INFO [IPC Server handler 8 on 56139] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741891_1067 127.0.0.1:53553 127.0.0.1:54793 127.0.0.1:57095 2018-06-21 23:23:38,298 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(4552): Deleted recovered.edits file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/36d0a953419e9f81636aebf73e7ed5df/recovered.edits/0000000000000000007 2018-06-21 23:23:38,298 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(931): stopping wal replay for 36d0a953419e9f81636aebf73e7ed5df 2018-06-21 23:23:38,298 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(943): Cleaning up temporary data for 36d0a953419e9f81636aebf73e7ed5df 2018-06-21 23:23:38,298 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/d4ac3b44d02d68507d6317e201672011/family/505db391b1184e37a9da64c4286f9b35, entries=40, sequenceid=7, filesize=6.3 K 2018-06-21 23:23:38,298 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(976): writing seq id for c793c2c3948aad84865d30ebf71a529e 2018-06-21 23:23:38,298 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(2769): Finished flush of dataSize ~2.13 KB/2184, heapSize ~4.59 KB/4696, currentSize=0 B/0 for d4ac3b44d02d68507d6317e201672011 in 118ms, sequenceid=7, compaction requested=false; wal=null 2018-06-21 23:23:38,299 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(931): stopping wal replay for 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:38,300 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(943): Cleaning up temporary data for 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:38,300 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(954): Cleaning up detritus for 36d0a953419e9f81636aebf73e7ed5df 2018-06-21 23:23:38,301 INFO [IPC Server handler 2 on 56139] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741886_1062 127.0.0.1:54793 127.0.0.1:57095 127.0.0.1:42074 2018-06-21 23:23:38,301 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(954): Cleaning up detritus for 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:38,304 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(4552): Deleted recovered.edits file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/d4ac3b44d02d68507d6317e201672011/recovered.edits/0000000000000000007 2018-06-21 23:23:38,304 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(976): writing seq id for fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:38,304 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(931): stopping wal replay for d4ac3b44d02d68507d6317e201672011 2018-06-21 23:23:38,304 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(943): Cleaning up temporary data for d4ac3b44d02d68507d6317e201672011 2018-06-21 23:23:38,307 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(976): writing seq id for 36d0a953419e9f81636aebf73e7ed5df 2018-06-21 23:23:38,307 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(976): writing seq id for 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:38,323 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(954): Cleaning up detritus for d4ac3b44d02d68507d6317e201672011 2018-06-21 23:23:38,324 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=5 2018-06-21 23:23:38,327 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/recovered.edits/13.seqid, newMaxSeqId=13, maxSeqId=10 2018-06-21 23:23:38,328 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/c793c2c3948aad84865d30ebf71a529e/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=1 2018-06-21 23:23:38,327 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/36d0a953419e9f81636aebf73e7ed5df/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=1 2018-06-21 23:23:38,328 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(980): Opened 0dfb3ebb8c759d632ec793ac0387ec95; next sequenceid=10 2018-06-21 23:23:38,328 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(987): Running coprocessor post-open hooks for 0dfb3ebb8c759d632ec793ac0387ec95 2018-06-21 23:23:38,328 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(976): writing seq id for d4ac3b44d02d68507d6317e201672011 2018-06-21 23:23:38,329 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(980): Opened 36d0a953419e9f81636aebf73e7ed5df; next sequenceid=8 2018-06-21 23:23:38,329 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(987): Running coprocessor post-open hooks for 36d0a953419e9f81636aebf73e7ed5df 2018-06-21 23:23:38,329 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(980): Opened fee67ae5d4f2b2be35fa516877a8d173; next sequenceid=14 2018-06-21 23:23:38,329 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(987): Running coprocessor post-open hooks for fee67ae5d4f2b2be35fa516877a8d173 2018-06-21 23:23:38,329 INFO [PostOpenDeployTasks:0dfb3ebb8c759d632ec793ac0387ec95] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95. 2018-06-21 23:23:38,329 INFO [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(980): Opened c793c2c3948aad84865d30ebf71a529e; next sequenceid=8 2018-06-21 23:23:38,330 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] regionserver.HRegion(987): Running coprocessor post-open hooks for c793c2c3948aad84865d30ebf71a529e 2018-06-21 23:23:38,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=10, pid=63, ppid=55, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=0dfb3ebb8c759d632ec793ac0387ec95; rit=OPENING, location=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:38,331 DEBUG [PEWorker-6] assignment.RegionTransitionProcedure(354): Finishing pid=63, ppid=55, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=0dfb3ebb8c759d632ec793ac0387ec95; rit=OPENING, location=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:38,331 INFO [PostOpenDeployTasks:c793c2c3948aad84865d30ebf71a529e] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,eee,1529623399386.c793c2c3948aad84865d30ebf71a529e. 2018-06-21 23:23:38,331 INFO [PostOpenDeployTasks:fee67ae5d4f2b2be35fa516877a8d173] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173. 2018-06-21 23:23:38,332 INFO [PEWorker-6] assignment.RegionStateStore(198): pid=63 updating hbase:meta row=0dfb3ebb8c759d632ec793ac0387ec95, regionState=OPEN, openSeqNum=10, regionLocation=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:38,336 DEBUG [PostOpenDeployTasks:0dfb3ebb8c759d632ec793ac0387ec95] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95. 2018-06-21 23:23:38,336 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] handler.OpenRegionHandler(128): Opened multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95. on asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:38,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=14, pid=59, ppid=55, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=fee67ae5d4f2b2be35fa516877a8d173; rit=OPENING, location=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:38,337 INFO [PostOpenDeployTasks:36d0a953419e9f81636aebf73e7ed5df] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,ppp,1529623399386.36d0a953419e9f81636aebf73e7ed5df. 2018-06-21 23:23:38,337 DEBUG [PEWorker-16] assignment.RegionTransitionProcedure(354): Finishing pid=59, ppid=55, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=fee67ae5d4f2b2be35fa516877a8d173; rit=OPENING, location=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:38,337 INFO [PEWorker-16] assignment.RegionStateStore(198): pid=59 updating hbase:meta row=fee67ae5d4f2b2be35fa516877a8d173, regionState=OPEN, openSeqNum=14, regionLocation=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:38,337 DEBUG [PostOpenDeployTasks:fee67ae5d4f2b2be35fa516877a8d173] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173. 2018-06-21 23:23:38,342 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] handler.OpenRegionHandler(128): Opened multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173. on asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:38,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=8, pid=60, ppid=55, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=36d0a953419e9f81636aebf73e7ed5df; rit=OPENING, location=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:38,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=8, pid=57, ppid=55, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=c793c2c3948aad84865d30ebf71a529e; rit=OPENING, location=asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:38,343 DEBUG [PEWorker-1] assignment.RegionTransitionProcedure(354): Finishing pid=60, ppid=55, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=36d0a953419e9f81636aebf73e7ed5df; rit=OPENING, location=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:38,343 DEBUG [PostOpenDeployTasks:36d0a953419e9f81636aebf73e7ed5df] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,ppp,1529623399386.36d0a953419e9f81636aebf73e7ed5df. 2018-06-21 23:23:38,343 INFO [PEWorker-1] assignment.RegionStateStore(198): pid=60 updating hbase:meta row=36d0a953419e9f81636aebf73e7ed5df, regionState=OPEN, openSeqNum=8, regionLocation=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:38,343 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] handler.OpenRegionHandler(128): Opened multi_test_table,ppp,1529623399386.36d0a953419e9f81636aebf73e7ed5df. on asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:38,343 DEBUG [PEWorker-11] assignment.RegionTransitionProcedure(354): Finishing pid=57, ppid=55, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=c793c2c3948aad84865d30ebf71a529e; rit=OPENING, location=asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:38,343 INFO [PEWorker-11] assignment.RegionStateStore(198): pid=57 updating hbase:meta row=c793c2c3948aad84865d30ebf71a529e, regionState=OPEN, openSeqNum=8, regionLocation=asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:38,343 DEBUG [PostOpenDeployTasks:c793c2c3948aad84865d30ebf71a529e] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,eee,1529623399386.c793c2c3948aad84865d30ebf71a529e. 2018-06-21 23:23:38,346 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-0] handler.OpenRegionHandler(128): Opened multi_test_table,eee,1529623399386.c793c2c3948aad84865d30ebf71a529e. on asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:38,346 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/d4ac3b44d02d68507d6317e201672011/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=1 2018-06-21 23:23:38,347 INFO [PEWorker-3] procedure2.ProcedureExecutor(1266): Finished pid=63, ppid=55, state=SUCCESS; AssignProcedure table=multi_test_table, region=0dfb3ebb8c759d632ec793ac0387ec95 in 721msec 2018-06-21 23:23:38,347 INFO [PEWorker-8] procedure2.ProcedureExecutor(1266): Finished pid=60, ppid=55, state=SUCCESS; AssignProcedure table=multi_test_table, region=36d0a953419e9f81636aebf73e7ed5df in 725msec 2018-06-21 23:23:38,347 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(980): Opened d4ac3b44d02d68507d6317e201672011; next sequenceid=8 2018-06-21 23:23:38,347 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(987): Running coprocessor post-open hooks for d4ac3b44d02d68507d6317e201672011 2018-06-21 23:23:38,347 INFO [PEWorker-10] procedure2.ProcedureExecutor(1266): Finished pid=57, ppid=55, state=SUCCESS; AssignProcedure table=multi_test_table, region=c793c2c3948aad84865d30ebf71a529e in 733msec 2018-06-21 23:23:38,347 INFO [PEWorker-7] procedure2.ProcedureExecutor(1266): Finished pid=59, ppid=55, state=SUCCESS; AssignProcedure table=multi_test_table, region=fee67ae5d4f2b2be35fa516877a8d173 in 722msec 2018-06-21 23:23:38,349 INFO [PostOpenDeployTasks:d4ac3b44d02d68507d6317e201672011] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,sss,1529623399386.d4ac3b44d02d68507d6317e201672011. 2018-06-21 23:23:38,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=8, pid=61, ppid=55, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=d4ac3b44d02d68507d6317e201672011; rit=OPENING, location=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:38,352 DEBUG [PEWorker-12] assignment.RegionTransitionProcedure(354): Finishing pid=61, ppid=55, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=d4ac3b44d02d68507d6317e201672011; rit=OPENING, location=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:38,352 DEBUG [PostOpenDeployTasks:d4ac3b44d02d68507d6317e201672011] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,sss,1529623399386.d4ac3b44d02d68507d6317e201672011. 2018-06-21 23:23:38,352 INFO [PEWorker-12] assignment.RegionStateStore(198): pid=61 updating hbase:meta row=d4ac3b44d02d68507d6317e201672011, regionState=OPEN, openSeqNum=8, regionLocation=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:38,352 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] handler.OpenRegionHandler(128): Opened multi_test_table,sss,1529623399386.d4ac3b44d02d68507d6317e201672011. on asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:38,525 INFO [PEWorker-2] procedure2.ProcedureExecutor(1266): Finished pid=62, ppid=55, state=SUCCESS; AssignProcedure table=multi_test_table, region=40503e5e9a757a4ccf579bf29b8d78c6 in 676msec 2018-06-21 23:23:38,526 INFO [PEWorker-1] procedure2.ProcedureExecutor(1266): Finished pid=60, ppid=55, state=SUCCESS; AssignProcedure table=multi_test_table, region=36d0a953419e9f81636aebf73e7ed5df in 725msec 2018-06-21 23:23:38,526 INFO [PEWorker-12] procedure2.ProcedureExecutor(1266): Finished pid=61, ppid=55, state=SUCCESS; AssignProcedure table=multi_test_table, region=d4ac3b44d02d68507d6317e201672011 in 733msec 2018-06-21 23:23:38,525 INFO [PEWorker-16] procedure2.ProcedureExecutor(1266): Finished pid=59, ppid=55, state=SUCCESS; AssignProcedure table=multi_test_table, region=fee67ae5d4f2b2be35fa516877a8d173 in 722msec 2018-06-21 23:23:38,525 INFO [PEWorker-6] procedure2.ProcedureExecutor(1266): Finished pid=63, ppid=55, state=SUCCESS; AssignProcedure table=multi_test_table, region=0dfb3ebb8c759d632ec793ac0387ec95 in 721msec 2018-06-21 23:23:38,526 INFO [PEWorker-11] procedure2.ProcedureExecutor(1266): Finished pid=57, ppid=55, state=SUCCESS; AssignProcedure table=multi_test_table, region=c793c2c3948aad84865d30ebf71a529e in 733msec 2018-06-21 23:23:38,611 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=2.29 KB at sequenceid=7 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/91ab16f39c84b448bdc2e5a1e1fc801b/.tmp/family/525f9ff8aa2b47b5bbb1545dd94b1c9f 2018-06-21 23:23:38,622 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/91ab16f39c84b448bdc2e5a1e1fc801b/.tmp/family/525f9ff8aa2b47b5bbb1545dd94b1c9f as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/91ab16f39c84b448bdc2e5a1e1fc801b/family/525f9ff8aa2b47b5bbb1545dd94b1c9f 2018-06-21 23:23:38,631 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/91ab16f39c84b448bdc2e5a1e1fc801b/family/525f9ff8aa2b47b5bbb1545dd94b1c9f, entries=44, sequenceid=7, filesize=6.4 K 2018-06-21 23:23:38,631 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(2769): Finished flush of dataSize ~2.29 KB/2340, heapSize ~5.02 KB/5144, currentSize=0 B/0 for 91ab16f39c84b448bdc2e5a1e1fc801b in 456ms, sequenceid=7, compaction requested=false; wal=null 2018-06-21 23:23:38,632 INFO [IPC Server handler 5 on 56139] blockmanagement.BlockManager(1169): BLOCK* addToInvalidates: blk_1073741889_1065 127.0.0.1:42074 127.0.0.1:53553 127.0.0.1:33402 2018-06-21 23:23:38,632 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(4552): Deleted recovered.edits file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/91ab16f39c84b448bdc2e5a1e1fc801b/recovered.edits/0000000000000000007 2018-06-21 23:23:38,633 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(931): stopping wal replay for 91ab16f39c84b448bdc2e5a1e1fc801b 2018-06-21 23:23:38,633 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(943): Cleaning up temporary data for 91ab16f39c84b448bdc2e5a1e1fc801b 2018-06-21 23:23:38,634 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(954): Cleaning up detritus for 91ab16f39c84b448bdc2e5a1e1fc801b 2018-06-21 23:23:38,636 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(976): writing seq id for 91ab16f39c84b448bdc2e5a1e1fc801b 2018-06-21 23:23:38,640 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/91ab16f39c84b448bdc2e5a1e1fc801b/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=1 2018-06-21 23:23:38,641 INFO [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(980): Opened 91ab16f39c84b448bdc2e5a1e1fc801b; next sequenceid=8 2018-06-21 23:23:38,641 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] regionserver.HRegion(987): Running coprocessor post-open hooks for 91ab16f39c84b448bdc2e5a1e1fc801b 2018-06-21 23:23:38,643 INFO [PostOpenDeployTasks:91ab16f39c84b448bdc2e5a1e1fc801b] regionserver.HRegionServer(2187): Post open deploy tasks for multi_test_table,hhh,1529623399386.91ab16f39c84b448bdc2e5a1e1fc801b. 2018-06-21 23:23:38,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=8, pid=58, ppid=55, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=multi_test_table, region=91ab16f39c84b448bdc2e5a1e1fc801b; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:38,647 DEBUG [PEWorker-9] assignment.RegionTransitionProcedure(354): Finishing pid=58, ppid=55, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=multi_test_table, region=91ab16f39c84b448bdc2e5a1e1fc801b; rit=OPENING, location=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:38,647 DEBUG [PostOpenDeployTasks:91ab16f39c84b448bdc2e5a1e1fc801b] regionserver.HRegionServer(2211): Finished post open deploy task for multi_test_table,hhh,1529623399386.91ab16f39c84b448bdc2e5a1e1fc801b. 2018-06-21 23:23:38,647 INFO [PEWorker-9] assignment.RegionStateStore(198): pid=58 updating hbase:meta row=91ab16f39c84b448bdc2e5a1e1fc801b, regionState=OPEN, openSeqNum=8, regionLocation=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:38,647 DEBUG [RS_OPEN_REGION-regionserver/asf911:0-1] handler.OpenRegionHandler(128): Opened multi_test_table,hhh,1529623399386.91ab16f39c84b448bdc2e5a1e1fc801b. on asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:38,748 INFO [PEWorker-9] procedure2.ProcedureExecutor(1635): Finished subprocedure(s) of pid=55, state=RUNNABLE:SERVER_CRASH_HANDLE_RIT2; ServerCrashProcedure server=asf911.gq1.ygridcore.net,40767,1529623390148, splitWal=true, meta=true; resume parent processing. 2018-06-21 23:23:38,749 INFO [PEWorker-9] procedure2.ProcedureExecutor(1266): Finished pid=58, ppid=55, state=SUCCESS; AssignProcedure table=multi_test_table, region=91ab16f39c84b448bdc2e5a1e1fc801b in 1.0280sec 2018-06-21 23:23:38,873 INFO [PEWorker-14] procedure2.ProcedureExecutor(1266): Finished pid=55, state=SUCCESS; ServerCrashProcedure server=asf911.gq1.ygridcore.net,40767,1529623390148, splitWal=true, meta=true in 4.1990sec 2018-06-21 23:23:38,897 INFO [RS-EventLoopGroup-10-16] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-06-21 23:23:38,897 INFO [RS-EventLoopGroup-10-16] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 52381 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:38,904 INFO [Time-limited test] client.TestMultiParallel(327): validating loaded data 2018-06-21 23:23:38,904 INFO [Time-limited test] client.TestMultiParallel(755): Validating data on multi_test_table;hconnection-0xe1c5692 2018-06-21 23:23:38,976 INFO [Time-limited test] client.TestMultiParallel(792): Validating data on multi_test_table;hconnection-0xe1c5692 successfully! 2018-06-21 23:23:38,977 INFO [Time-limited test] hbase.LocalHBaseCluster(270): Not alive RS:0;asf911:59970 2018-06-21 23:23:38,977 INFO [Time-limited test] hbase.LocalHBaseCluster(270): Not alive RS:1;asf911:40767 2018-06-21 23:23:38,977 INFO [Time-limited test] client.TestMultiParallel(335): Count=1, Alive=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:38,977 INFO [Time-limited test] client.TestMultiParallel(335): Count=2, Alive=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:38,977 INFO [Time-limited test] client.TestMultiParallel(335): Count=3, Alive=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:38,977 INFO [Time-limited test] client.TestMultiParallel(335): Count=4, Alive=asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:38,977 INFO [Time-limited test] client.TestMultiParallel(337): Count=4 2018-06-21 23:23:38,978 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [15,000] milli-secs(wait.for.ratio=[1]) 2018-06-21 23:23:38,980 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [15,000] milli-secs(wait.for.ratio=[1]) 2018-06-21 23:23:38,980 INFO [Time-limited test] client.TestMultiParallel(354): done 2018-06-21 23:23:39,038 INFO [Time-limited test] hbase.ResourceChecker(172): after: client.TestMultiParallel#testFlushCommitsWithAbort Thread=668 (was 639) Potentially hanging thread: RpcServer.priority.FPBQ.Fifo.handler=2,queue=0,port=34033 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Potentially hanging thread: RSProcedureDispatcher-pool3-t30 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_714311547_23 at /127.0.0.1:60774 [Waiting for operation #9] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t46 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS_CLOSE_REGION-master/asf911:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t37 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-shared-pool15-t34 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-10-8 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-10-1 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t26 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS_LOG_REPLAY_OPS-regionserver/asf911:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-10-7 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=34033 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1552870040_23 at /127.0.0.1:57059 [Waiting for operation #18] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-shared-pool15-t35 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t35 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: ReadOnlyZKClient-localhost:58989@0x0160fc1e-SendThread(localhost:58989) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t48 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-10-5 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t42 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_321370780_23 at /127.0.0.1:45147 [Receiving block BP-1812449274-67.195.81.155-1529623383006:blk_1073741883_1059] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:200) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:503) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:903) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:805) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:253) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t32 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t19 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-9-17 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-1-10 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1552870040_23 at /127.0.0.1:33001 [Receiving block BP-1812449274-67.195.81.155-1529623383006:blk_1073741879_1055] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:200) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:503) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:903) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:805) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:253) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-10-9 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-9-16 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-10-16 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: PacketResponder: BP-1812449274-67.195.81.155-1529623383006:blk_1073741883_1059, type=LAST_IN_PIPELINE, downstreams=0:[] java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1238) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1309) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-10-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: ReadOnlyZKClient-localhost:58989@0x0160fc1e-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) Potentially hanging thread: RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=34033 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Potentially hanging thread: RS-EventLoopGroup-10-15 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t22 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-shared-pool15-t39 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_501136618_23 at /127.0.0.1:45190 [Waiting for operation #6] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: PacketResponder: BP-1812449274-67.195.81.155-1529623383006:blk_1073741879_1055, type=LAST_IN_PIPELINE, downstreams=0:[] java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1238) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1309) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t49 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t36 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_400506704_23 at /127.0.0.1:33257 [Waiting for operation #11] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: LeaseRenewer:jenkins.hfs.5@localhost:56139 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:444) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:304) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t34 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t52 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-10-6 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t51 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t38 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t40 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS:5;asf911:34033 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:92) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:56) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1011) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:183) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:129) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:167) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:360) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:164) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-10-12 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data10/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_714311547_23 at /127.0.0.1:60739 [Waiting for operation #12] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t23 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t31 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1552870040_23 at /127.0.0.1:44805 [Receiving block BP-1812449274-67.195.81.155-1529623383006:blk_1073741879_1055] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:200) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:503) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:903) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:805) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:253) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RpcServer.priority.FPBQ.Fifo.handler=1,queue=0,port=34033 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t44 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: IPC Client (443388289) connection to localhost/127.0.0.1:56139 from jenkins.hfs.5 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:934) org.apache.hadoop.ipc.Client$Connection.run(Client.java:979) Potentially hanging thread: RS_OPEN_META-master/asf911:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RpcServer.priority.FPBQ.Fifo.handler=0,queue=0,port=34033 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t33 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-shared-pool15-t38 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=34033 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t62 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t41 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t35 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t57 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-shared-pool15-t36 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t32 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: SplitLogWorker-asf911:34033 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination.taskLoop(ZkSplitLogWorkerCoordination.java:461) org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:144) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-10-4 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t47 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_321370780_23 at /127.0.0.1:33246 [Receiving block BP-1812449274-67.195.81.155-1529623383006:blk_1073741883_1059] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:200) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:503) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:903) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:805) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:253) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: PacketResponder: BP-1812449274-67.195.81.155-1529623383006:blk_1073741883_1059, type=LAST_IN_PIPELINE, downstreams=0:[] java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1238) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1309) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t33 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_501136618_23 at /127.0.0.1:45226 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t53 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-10-11 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t50 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RpcServer.priority.FPBQ.Fifo.handler=3,queue=0,port=34033 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=34033 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t43 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t61 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-10-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t60 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=34033 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Potentially hanging thread: hconnection-0xe1c5692-shared-pool15-t40 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t58 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-shared-pool15-t37 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: PacketResponder: BP-1812449274-67.195.81.155-1529623383006:blk_1073741879_1055, type=LAST_IN_PIPELINE, downstreams=0:[] java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1238) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1309) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_501136618_23 at /127.0.0.1:45137 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t39 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_501136618_23 at /127.0.0.1:45152 [Waiting for operation #7] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data5/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS:5;asf911:34033-longCompactions-1529623407904 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:106) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: ReadOnlyZKClient-localhost:58989@0x0160fc1e sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:313) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$66/1430146525.run(Unknown Source) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=34033 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data9/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t20 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=34033 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Potentially hanging thread: RS_CLOSE_REGION-master/asf911:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t28 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: PacketResponder: BP-1812449274-67.195.81.155-1529623383006:blk_1073741879_1055, type=LAST_IN_PIPELINE, downstreams=0:[] java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1238) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1309) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=34033 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t63 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data7/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-10-14 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t21 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_400506704_23 at /127.0.0.1:45015 [Waiting for operation #17] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS_CLOSE_REGION-master/asf911:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=34033 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t59 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t56 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t55 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-10-10 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t24 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-shared-pool15-t31 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t34 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1552870040_23 at /127.0.0.1:56813 [Receiving block BP-1812449274-67.195.81.155-1529623383006:blk_1073741879_1055] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:200) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:503) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:903) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:805) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:253) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t54 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t29 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t65 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: PacketResponder: BP-1812449274-67.195.81.155-1529623383006:blk_1073741883_1059, type=LAST_IN_PIPELINE, downstreams=0:[] java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:502) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.waitForAckHead(BlockReceiver.java:1238) org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1309) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t27 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_321370780_23 at /127.0.0.1:45050 [Receiving block BP-1812449274-67.195.81.155-1529623383006:blk_1073741883_1059] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) java.io.DataInputStream.read(DataInputStream.java:149) org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:200) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134) org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:503) org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:903) org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:805) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:253) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-271332707_23 at /127.0.0.1:57102 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:229) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t45 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data8/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-9-15 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t64 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: hconnection-0xe1c5692-metaLookup-shared--pool16-t66 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34033 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.Semaphore.acquire(Semaphore.java:312) org.apache.hadoop.hbase.ipc.FastPathBalancedQueueRpcExecutor$FastPathHandler.getCallRunner(FastPathBalancedQueueRpcExecutor.java:104) org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data6/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: Default-IPC-NioEventLoopGroup-9-18 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:753) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:409) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RS-EventLoopGroup-10-13 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:122) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:235) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:252) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: RSProcedureDispatcher-pool3-t17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Potentially hanging thread: asf911:34033Replication Statistics #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) - Thread LEAK? -, OpenFileDescriptor=2456 (was 2437) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=689 (was 601) - SystemLoadAverage LEAK? -, ProcessCount=271 (was 269) - ProcessCount LEAK? -, AvailableMemoryMB=13350 (was 13466) 2018-06-21 23:23:39,039 WARN [Time-limited test] hbase.ResourceChecker(135): Thread=668 is superior to 500 2018-06-21 23:23:39,039 WARN [Time-limited test] hbase.ResourceChecker(135): OpenFileDescriptor=2456 is superior to 1024 2018-06-21 23:23:39,082 INFO [Time-limited test] hbase.ResourceChecker(148): before: client.TestMultiParallel#testActiveThreadsCount Thread=668, OpenFileDescriptor=2456, MaxFileDescriptor=60000, SystemLoadAverage=689, ProcessCount=271, AvailableMemoryMB=13350 2018-06-21 23:23:39,082 WARN [Time-limited test] hbase.ResourceChecker(135): Thread=668 is superior to 500 2018-06-21 23:23:39,082 WARN [Time-limited test] hbase.ResourceChecker(135): OpenFileDescriptor=2456 is superior to 1024 2018-06-21 23:23:39,082 INFO [Time-limited test] client.TestMultiParallel(117): before 2018-06-21 23:23:39,082 INFO [Time-limited test] hbase.LocalHBaseCluster(270): Not alive RS:1;asf911:40767 2018-06-21 23:23:39,108 INFO [Time-limited test] client.ConnectionUtils(122): regionserver/asf911:0 server-side Connection retries=45 2018-06-21 23:23:39,108 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=50, handlerCount=5 2018-06-21 23:23:39,109 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated priority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=60, handlerCount=6 2018-06-21 23:23:39,109 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2018-06-21 23:23:39,109 INFO [Time-limited test] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-06-21 23:23:39,109 INFO [Time-limited test] io.ByteBufferPool(83): Created with bufferSize=64 KB and maxPoolSize=320 B 2018-06-21 23:23:39,110 INFO [Time-limited test] ipc.NettyRpcServer(109): Bind to /67.195.81.155:50612 2018-06-21 23:23:39,111 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=24, currentSize=803.04 KB, freeSize=994.82 MB, maxSize=995.60 MB, heapSize=803.04 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:39,111 INFO [Time-limited test] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=24, currentSize=803.04 KB, freeSize=994.82 MB, maxSize=995.60 MB, heapSize=803.04 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:39,112 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-06-21 23:23:39,114 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-06-21 23:23:39,115 INFO [Time-limited test] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:50612 connecting to ZooKeeper ensemble=localhost:58989 2018-06-21 23:23:39,133 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:506120x0, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-06-21 23:23:39,135 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(543): regionserver:50612-0x16424a89dd40011 connected 2018-06-21 23:23:39,135 DEBUG [Time-limited test] zookeeper.ZKUtil(355): regionserver:50612-0x16424a89dd40011, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-06-21 23:23:39,137 DEBUG [Time-limited test] zookeeper.ZKUtil(355): regionserver:50612-0x16424a89dd40011, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/running 2018-06-21 23:23:39,139 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=5 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=50612 2018-06-21 23:23:39,140 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=6 with threadPrefix=priority.FPBQ.Fifo, numCallQueues=1, port=50612 2018-06-21 23:23:39,141 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=50612 2018-06-21 23:23:39,148 INFO [RS:5;asf911:50612] regionserver.HRegionServer(874): ClusterId : 2ed7179d-84c9-432a-b6b5-556343aa7690 2018-06-21 23:23:39,148 DEBUG [RS:5;asf911:50612] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initializing 2018-06-21 23:23:39,184 DEBUG [RS:5;asf911:50612] procedure.RegionServerProcedureManagerHost(47): Procedure flush-table-proc initialized 2018-06-21 23:23:39,184 DEBUG [RS:5;asf911:50612] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initializing 2018-06-21 23:23:39,192 DEBUG [RS:5;asf911:50612] procedure.RegionServerProcedureManagerHost(47): Procedure online-snapshot initialized 2018-06-21 23:23:39,195 INFO [RS:5;asf911:50612] zookeeper.ReadOnlyZKClient(139): Connect 0x70e8353a to localhost:58989 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-06-21 23:23:39,243 DEBUG [RS:5;asf911:50612] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e2fc557, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-06-21 23:23:39,243 DEBUG [RS:5;asf911:50612] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a105e4a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=asf911.gq1.ygridcore.net/67.195.81.155:0 2018-06-21 23:23:39,244 DEBUG [RS:5;asf911:50612] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:5;asf911:50612 2018-06-21 23:23:39,244 INFO [RS:5;asf911:50612] regionserver.RegionServerCoprocessorHost(67): System coprocessor loading is enabled 2018-06-21 23:23:39,244 INFO [RS:5;asf911:50612] regionserver.RegionServerCoprocessorHost(68): Table coprocessor loading is enabled 2018-06-21 23:23:39,245 INFO [RS:5;asf911:50612] regionserver.HRegionServer(2600): reportForDuty to master=asf911.gq1.ygridcore.net,55146,1529623388534 with port=50612, startcode=1529623419106 2018-06-21 23:23:39,247 INFO [RS-EventLoopGroup-1-11] ipc.ServerRpcConnection(528): Auth successful for jenkins.hfs.6 (auth:SIMPLE) 2018-06-21 23:23:39,248 INFO [RS-EventLoopGroup-1-11] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 45325 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:39,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=55146] master.ServerManager(439): Registering regionserver=asf911.gq1.ygridcore.net,50612,1529623419106 2018-06-21 23:23:39,249 DEBUG [RS:5;asf911:50612] regionserver.HRegionServer(1505): Config from master: hbase.rootdir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba 2018-06-21 23:23:39,250 DEBUG [RS:5;asf911:50612] regionserver.HRegionServer(1505): Config from master: fs.defaultFS=hdfs://localhost:56139 2018-06-21 23:23:39,250 DEBUG [RS:5;asf911:50612] regionserver.HRegionServer(1505): Config from master: hbase.master.info.port=-1 2018-06-21 23:23:39,283 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:39,283 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:39,283 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:39,283 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:39,283 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:39,284 DEBUG [RS:5;asf911:50612] zookeeper.ZKUtil(355): regionserver:50612-0x16424a89dd40011, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,50612,1529623419106 2018-06-21 23:23:39,284 WARN [RS:5;asf911:50612] hbase.ZNodeClearer(63): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-06-21 23:23:39,284 INFO [RS:5;asf911:50612] wal.WALFactory(137): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-06-21 23:23:39,284 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:39,284 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:39,284 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:39,284 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:39,284 DEBUG [RS:5;asf911:50612] regionserver.HRegionServer(1811): logDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,50612,1529623419106 2018-06-21 23:23:39,285 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:39,285 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:39,285 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:39,285 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:39,285 INFO [RegionServerTracker-0] master.RegionServerTracker(170): RegionServer ephemeral node created, adding [asf911.gq1.ygridcore.net,50612,1529623419106] 2018-06-21 23:23:39,286 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,50612,1529623419106 2018-06-21 23:23:39,286 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,50612,1529623419106 2018-06-21 23:23:39,285 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:39,286 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,50612,1529623419106 2018-06-21 23:23:39,286 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:39,289 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:39,289 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,50612,1529623419106 2018-06-21 23:23:39,289 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:39,291 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:39,291 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:39,291 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,50612,1529623419106 2018-06-21 23:23:39,291 INFO [Thread-927] assignment.AssignmentManager(492): Async MOVE of 1588230740 to newer Server=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:39,291 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:39,291 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:39,294 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:39,294 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:39,294 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:39,294 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:39,294 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:39,294 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:39,294 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:39,295 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:39,295 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:39,296 DEBUG [RS:5;asf911:50612] zookeeper.ZKUtil(355): regionserver:50612-0x16424a89dd40011, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:39,296 DEBUG [RS:5;asf911:50612] zookeeper.ZKUtil(355): regionserver:50612-0x16424a89dd40011, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:39,297 DEBUG [RS:5;asf911:50612] zookeeper.ZKUtil(355): regionserver:50612-0x16424a89dd40011, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,50612,1529623419106 2018-06-21 23:23:39,298 DEBUG [RS:5;asf911:50612] zookeeper.ZKUtil(355): regionserver:50612-0x16424a89dd40011, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:39,298 DEBUG [RS:5;asf911:50612] zookeeper.ZKUtil(355): regionserver:50612-0x16424a89dd40011, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:39,299 DEBUG [RS:5;asf911:50612] zookeeper.ZKUtil(355): regionserver:50612-0x16424a89dd40011, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:39,300 DEBUG [RS:5;asf911:50612] regionserver.Replication(132): Replication stats-in-log period=300 seconds 2018-06-21 23:23:39,300 INFO [RS:5;asf911:50612] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-06-21 23:23:39,303 INFO [RS:5;asf911:50612] regionserver.MemStoreFlusher(133): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-06-21 23:23:39,303 INFO [RS:5;asf911:50612] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/second, lower bound 10.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2018-06-21 23:23:39,304 INFO [RS:5;asf911:50612] regionserver.HRegionServer$CompactionChecker(1705): CompactionChecker runs every PT1S 2018-06-21 23:23:39,307 DEBUG [RS:5;asf911:50612] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:39,307 DEBUG [RS:5;asf911:50612] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-regionserver/asf911:0, corePoolSize=1, maxPoolSize=1 2018-06-21 23:23:39,307 DEBUG [RS:5;asf911:50612] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:39,308 DEBUG [RS:5;asf911:50612] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:39,308 DEBUG [RS:5;asf911:50612] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-regionserver/asf911:0, corePoolSize=1, maxPoolSize=1 2018-06-21 23:23:39,308 DEBUG [RS:5;asf911:50612] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/asf911:0, corePoolSize=2, maxPoolSize=2 2018-06-21 23:23:39,308 DEBUG [RS:5;asf911:50612] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/asf911:0, corePoolSize=10, maxPoolSize=10 2018-06-21 23:23:39,308 DEBUG [RS:5;asf911:50612] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/asf911:0, corePoolSize=3, maxPoolSize=3 2018-06-21 23:23:39,308 DEBUG [RS:5;asf911:50612] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-regionserver/asf911:0, corePoolSize=2, maxPoolSize=2 2018-06-21 23:23:39,336 INFO [SplitLogWorker-asf911:50612] regionserver.SplitLogWorker(136): SplitLogWorker asf911.gq1.ygridcore.net,50612,1529623419106 starting 2018-06-21 23:23:39,336 INFO [RS:5;asf911:50612] regionserver.HeapMemoryManager(210): Starting, tuneOn=false 2018-06-21 23:23:39,367 INFO [RS:5;asf911:50612] regionserver.HRegionServer(1546): Serving as asf911.gq1.ygridcore.net,50612,1529623419106, RpcServer on asf911.gq1.ygridcore.net/67.195.81.155:50612, sessionid=0x16424a89dd40011 2018-06-21 23:23:39,368 INFO [Time-limited test] hbase.HBaseTestingUtility(3080): Started new server=Thread[RS:5;asf911:50612,5,FailOnTimeoutGroup] 2018-06-21 23:23:39,368 DEBUG [RS:5;asf911:50612] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc starting 2018-06-21 23:23:39,368 DEBUG [RS:5;asf911:50612] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager asf911.gq1.ygridcore.net,50612,1529623419106 2018-06-21 23:23:39,368 DEBUG [RS:5;asf911:50612] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'asf911.gq1.ygridcore.net,50612,1529623419106' 2018-06-21 23:23:39,368 DEBUG [RS:5;asf911:50612] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-06-21 23:23:39,369 INFO [Time-limited test] master.HMaster(1534): Balancer plans size is 1, the balance interval is 300000 ms, and the max number regions in transition is 28 2018-06-21 23:23:39,369 INFO [Time-limited test] master.HMaster(1539): balance hri=1588230740, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:39,369 DEBUG [RS:5;asf911:50612] procedure.ZKProcedureMemberRpcs(155): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-06-21 23:23:39,371 DEBUG [RS:5;asf911:50612] procedure.RegionServerProcedureManagerHost(55): Procedure flush-table-proc started 2018-06-21 23:23:39,372 DEBUG [RS:5;asf911:50612] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot starting 2018-06-21 23:23:39,372 DEBUG [RS:5;asf911:50612] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager asf911.gq1.ygridcore.net,50612,1529623419106 2018-06-21 23:23:39,372 DEBUG [RS:5;asf911:50612] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'asf911.gq1.ygridcore.net,50612,1529623419106' 2018-06-21 23:23:39,372 DEBUG [RS:5;asf911:50612] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-06-21 23:23:39,372 DEBUG [RS:5;asf911:50612] procedure.ZKProcedureMemberRpcs(155): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-06-21 23:23:39,373 DEBUG [RS:5;asf911:50612] procedure.RegionServerProcedureManagerHost(55): Procedure online-snapshot started 2018-06-21 23:23:39,373 INFO [RS:5;asf911:50612] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-06-21 23:23:39,373 INFO [RS:5;asf911:50612] quotas.RegionServerSpaceQuotaManager(84): Quota support disabled, not starting space quota manager. 2018-06-21 23:23:39,513 DEBUG [Thread-927] procedure2.ProcedureExecutor(887): Stored pid=64, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=1588230740, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination= 2018-06-21 23:23:39,513 DEBUG [Time-limited test] procedure2.ProcedureExecutor(887): Stored pid=65, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=1588230740, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:39,515 INFO [PEWorker-15] procedure.MasterProcedureScheduler(696): pid=64, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=1588230740, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination= checking lock on 1588230740 2018-06-21 23:23:39,516 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [3,000] milli-secs(wait.for.ratio=[1]) 2018-06-21 23:23:39,516 INFO [PEWorker-15] procedure2.ProcedureExecutor(1516): Initialized subprocedures=[{pid=66, ppid=64, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=hbase:meta, region=1588230740, server=asf911.gq1.ygridcore.net,55146,1529623388534}] 2018-06-21 23:23:39,517 INFO [PEWorker-5] procedure.MasterProcedureScheduler(696): pid=65, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=1588230740, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,34033,1529623407773 checking lock on 1588230740 2018-06-21 23:23:39,518 DEBUG [PEWorker-5] procedure2.ProcedureExecutor(1250): LOCK_EVENT_WAIT pid=65, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=1588230740, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:39,594 INFO [PEWorker-15] procedure.MasterProcedureScheduler(696): pid=66, ppid=64, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=hbase:meta, region=1588230740, server=asf911.gq1.ygridcore.net,55146,1529623388534 checking lock on 1588230740 2018-06-21 23:23:39,594 INFO [PEWorker-15] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:39,608 INFO [PEWorker-15] assignment.RegionTransitionProcedure(241): Dispatch pid=66, ppid=64, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=hbase:meta, region=1588230740, server=asf911.gq1.ygridcore.net,55146,1529623388534; rit=CLOSING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:39,618 DEBUG [Time-limited test] hbase.HBaseTestingUtility(3284): Waiting until all regions of table multi_test_table get assigned. Timeout = 60000ms 2018-06-21 23:23:39,618 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2018-06-21 23:23:39,627 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=55146] regionserver.StoreScanner(1000): Switch to stream read (scanned=32773 bytes) of info 2018-06-21 23:23:39,647 INFO [Time-limited test] hbase.HBaseTestingUtility(3336): All regions for table multi_test_table assigned to meta. Checking AM states. 2018-06-21 23:23:39,648 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2018-06-21 23:23:39,648 INFO [Time-limited test] hbase.HBaseTestingUtility(3356): All regions for table multi_test_table assigned. 2018-06-21 23:23:39,648 INFO [Time-limited test] client.TestMultiParallel(133): before done 2018-06-21 23:23:39,649 INFO [Time-limited test] zookeeper.ReadOnlyZKClient(139): Connect 0x60e8eedd to localhost:58989 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-06-21 23:23:39,684 DEBUG [Time-limited test] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69770861, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-06-21 23:23:39,686 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,690 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"\\x00","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,691 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,694 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"bbb\\x01","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,695 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,698 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ccc\\x02","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,699 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,701 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ddd\\x03","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,702 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,704 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"eee\\x04","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,705 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,708 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"fff\\x05","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,710 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,713 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ggg\\x06","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,713 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,716 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"hhh\\x07","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,717 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,720 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"iii\\x08","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,721 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,724 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"jjj\\x09","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,725 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,728 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"kkk\\x0A","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,730 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,732 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"lll\\x0B","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,733 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,736 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"mmm\\x0C","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,737 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,740 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"nnn\\x0D","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,741 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,745 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ooo\\x0E","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,746 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,749 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ppp\\x0F","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,750 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,753 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"qqq\\x10","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,754 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,757 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"rrr\\x11","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,758 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,758 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(119): Fallback to compat rpc execution for serverName=asf911.gq1.ygridcore.net,55146,1529623388534 version=0 2018-06-21 23:23:39,766 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"sss\\x12","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,767 INFO [RSProcedureDispatcher-pool3-t37] regionserver.RSRpcServices(1602): Close 1588230740 without moving 2018-06-21 23:23:39,767 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,769 DEBUG [RS_CLOSE_META-master/asf911:0-0] regionserver.HRegion(1520): Closing 1588230740, disabling compactions & flushes 2018-06-21 23:23:39,778 DEBUG [RS_CLOSE_META-master/asf911:0-0] regionserver.HRegion(1560): Updates disabled for region hbase:meta,,1.1588230740 2018-06-21 23:23:39,778 INFO [RS_CLOSE_META-master/asf911:0-0] regionserver.HRegion(2560): Flushing 3/3 column families, dataSize=6.96 KB heapSize=10.63 KB 2018-06-21 23:23:39,779 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ttt\\x13","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,780 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,783 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"uuu\\x14","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,784 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,787 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"vvv\\x15","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,788 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,791 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"www\\x16","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,792 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,795 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"xxx\\x17","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,796 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,799 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"yyy\\x18","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,800 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,802 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"\\x19","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,803 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,805 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"bbb\\x1A","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,806 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,808 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ccc\\x1B","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,809 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,811 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ddd\\x1C","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,812 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,815 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"eee\\x1D","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,816 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,819 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"fff\\x1E","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,820 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,824 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ggg\\x1F","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,825 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,827 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741900_1076{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|FINALIZED]]} size 0 2018-06-21 23:23:39,828 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741900_1076{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|FINALIZED]]} size 0 2018-06-21 23:23:39,828 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741900_1076{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|FINALIZED], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|FINALIZED]]} size 0 2018-06-21 23:23:39,828 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"hhh ","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,829 INFO [RS_CLOSE_META-master/asf911:0-0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=6.96 KB at sequenceid=110 (bloomFilter=false), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/.tmp/info/5284050d0da44000946b8468c73dd88e 2018-06-21 23:23:39,829 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,829 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-06-21 23:23:39,832 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"iii!","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,833 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,836 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"jjj\"","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,837 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,840 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"kkk#","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,841 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,844 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"lll$","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,845 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,847 DEBUG [RS_CLOSE_META-master/asf911:0-0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/.tmp/info/5284050d0da44000946b8468c73dd88e as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/info/5284050d0da44000946b8468c73dd88e 2018-06-21 23:23:39,848 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"mmm%","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,849 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,858 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"nnn&","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,859 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,868 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ooo'","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,869 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,870 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ppp(","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,871 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,872 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"qqq)","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,873 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,873 INFO [RS_CLOSE_META-master/asf911:0-0] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/info/5284050d0da44000946b8468c73dd88e, entries=56, sequenceid=110, filesize=11.7 K 2018-06-21 23:23:39,874 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"rrr*","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,875 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,875 INFO [RS_CLOSE_META-master/asf911:0-0] regionserver.HRegion(2769): Finished flush of dataSize ~6.96 KB/7126, heapSize ~11.26 KB/11528, currentSize=0 B/0 for 1588230740 in 97ms, sequenceid=110, compaction requested=false 2018-06-21 23:23:39,876 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"sss+","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,883 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,885 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ttt,","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,885 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,887 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"uuu-","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,889 DEBUG [RS_CLOSE_META-master/asf911:0-0] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/recovered.edits/113.seqid, newMaxSeqId=113, maxSeqId=93 2018-06-21 23:23:39,889 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,890 DEBUG [RS_CLOSE_META-master/asf911:0-0] coprocessor.CoprocessorHost(288): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-06-21 23:23:39,891 INFO [RS_CLOSE_META-master/asf911:0-0] regionserver.HRegion(1674): Closed hbase:meta,,1.1588230740 2018-06-21 23:23:39,891 DEBUG [RS_CLOSE_META-master/asf911:0-0] assignment.RegionTransitionProcedure(264): Received report CLOSED seqId=-1, pid=66, ppid=64, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=hbase:meta, region=1588230740, server=asf911.gq1.ygridcore.net,55146,1529623388534; rit=CLOSING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:39,891 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"vvv.","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,891 DEBUG [RS_CLOSE_META-master/asf911:0-0] handler.CloseRegionHandler(124): Closed hbase:meta,,1.1588230740 2018-06-21 23:23:39,894 DEBUG [PEWorker-3] assignment.RegionTransitionProcedure(354): Finishing pid=66, ppid=64, state=RUNNABLE:REGION_TRANSITION_FINISH; UnassignProcedure table=hbase:meta, region=1588230740, server=asf911.gq1.ygridcore.net,55146,1529623388534; rit=CLOSING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:39,894 WARN [PEWorker-3] zookeeper.MetaTableLocator(449): Tried to set null ServerName in hbase:meta; skipping -- ServerName required 2018-06-21 23:23:39,895 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,897 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"www/","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,897 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,899 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"xxx0","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,900 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,901 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"yyy1","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,901 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,903 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"2","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,903 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,905 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"bbb3","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,905 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,906 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ccc4","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,907 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,908 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ddd5","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,908 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,909 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"eee6","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,910 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,910 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"fff7","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,911 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,912 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ggg8","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,912 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,914 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"hhh9","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,915 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,916 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"iii:","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,916 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,917 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"jjj;","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,918 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,919 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"kkk<","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,920 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,920 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"lll=","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,921 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,925 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"mmm>","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,926 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,927 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"nnn?","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,927 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,928 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ooo@","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,929 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,929 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"pppA","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,930 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,931 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"qqqB","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,931 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,932 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"rrrC","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,933 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,934 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"sssD","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,934 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,935 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"tttE","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,936 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,937 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"uuuF","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,937 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,939 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"vvvG","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,939 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,941 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"wwwH","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,941 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,942 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"xxxI","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,943 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,944 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"yyyJ","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,945 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,946 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"K","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,947 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,948 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"bbbL","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,948 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,950 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"cccM","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,950 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,952 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"dddN","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,952 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,953 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"eeeO","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,954 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,955 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"fffP","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,956 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,957 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"gggQ","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,958 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,959 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"hhhR","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,959 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,960 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"iiiS","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,961 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,962 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"jjjT","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,963 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,964 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"kkkU","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,965 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,966 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"lllV","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,967 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,968 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"mmmW","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,969 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,970 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"nnnX","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,970 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,971 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"oooY","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,972 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,973 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"pppZ","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,974 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,975 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"qqq[","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,976 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,977 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"rrr\\x5C","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,978 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,979 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"sss]","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,979 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,980 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ttt^","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,981 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,982 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"uuu_","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,983 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-06-21 23:23:39,983 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,985 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"vvv`","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,986 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,987 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"wwwa","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,988 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,989 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"xxxb","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,990 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,991 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"yyyc","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,992 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,993 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"d","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,993 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,994 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"bbbe","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,995 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,996 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"cccf","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,996 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,997 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"dddg","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:39,997 INFO [PEWorker-3] procedure2.ProcedureExecutor(1635): Finished subprocedure(s) of pid=64, state=RUNNABLE:MOVE_REGION_ASSIGN; MoveRegionProcedure hri=1588230740, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=; resume parent processing. 2018-06-21 23:23:39,997 INFO [PEWorker-3] procedure2.ProcedureExecutor(1266): Finished pid=66, ppid=64, state=SUCCESS; UnassignProcedure table=hbase:meta, region=1588230740, server=asf911.gq1.ygridcore.net,55146,1529623388534 in 379msec 2018-06-21 23:23:39,997 INFO [PEWorker-8] procedure2.ProcedureExecutor(1516): Initialized subprocedures=[{pid=67, ppid=64, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740}] 2018-06-21 23:23:39,998 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:39,999 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"eeeh","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,000 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,001 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"fffi","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,002 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,003 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"gggj","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,004 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,005 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"hhhk","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,006 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,007 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"iiil","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,009 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,010 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"jjjm","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,011 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,012 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"kkkn","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,013 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,014 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"lllo","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,015 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,016 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"mmmp","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,017 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,017 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"nnnq","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,018 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,019 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ooor","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,020 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,021 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ppps","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,022 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,023 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"qqqt","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,024 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,025 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"rrru","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,026 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,027 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"sssv","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,028 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,029 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"tttw","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,030 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,032 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"uuux","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,032 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,034 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"vvvy","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,035 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,036 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"wwwz","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,037 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,038 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"xxx{","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,039 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,040 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"yyy|","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,041 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,042 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"}","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,043 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,044 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"bbb~","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,045 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,046 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ccc\\x7F","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,047 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,047 INFO [PEWorker-8] procedure.MasterProcedureScheduler(696): pid=67, ppid=64, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740 checking lock on 1588230740 2018-06-21 23:23:40,048 INFO [PEWorker-8] assignment.AssignProcedure(213): Setting lastHost as the region location asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:40,048 INFO [PEWorker-8] assignment.AssignProcedure(218): Starting pid=67, ppid=64, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=asf911.gq1.ygridcore.net,55146,1529623388534; forceNewPlan=false, retain=true 2018-06-21 23:23:40,048 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ddd\\x80","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,049 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,050 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"eee\\x81","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,051 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,052 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"fff\\x82","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,053 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,054 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ggg\\x83","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,055 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,056 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"hhh\\x84","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,056 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,058 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"iii\\x85","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,058 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,059 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"jjj\\x86","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,060 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,061 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"kkk\\x87","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,062 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,063 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"lll\\x88","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,064 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,065 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"mmm\\x89","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,066 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,067 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"nnn\\x8A","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,068 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,069 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ooo\\x8B","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,070 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,071 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ppp\\x8C","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,072 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,073 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"qqq\\x8D","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,074 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,075 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"rrr\\x8E","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,076 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,077 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"sss\\x8F","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,078 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,079 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ttt\\x90","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,080 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,081 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"uuu\\x91","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,081 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,082 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"vvv\\x92","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,083 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,084 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"www\\x93","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,085 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,086 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"xxx\\x94","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,087 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,088 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"yyy\\x95","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,089 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,090 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"\\x96","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,091 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,092 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"bbb\\x97","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,093 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,094 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ccc\\x98","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,094 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,095 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ddd\\x99","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,096 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,097 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"eee\\x9A","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,098 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,099 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"fff\\x9B","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,100 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,101 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ggg\\x9C","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,102 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,103 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"hhh\\x9D","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,104 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,105 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"iii\\x9E","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,106 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,107 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"jjj\\x9F","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,108 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,109 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"kkk\\xA0","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,110 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,111 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"lll\\xA1","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,112 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,113 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"mmm\\xA2","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,113 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,114 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"nnn\\xA3","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,115 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,116 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ooo\\xA4","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,117 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,118 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ppp\\xA5","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,119 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,120 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"qqq\\xA6","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,121 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,122 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"rrr\\xA7","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,123 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,124 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"sss\\xA8","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,125 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,126 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ttt\\xA9","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,126 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,127 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"uuu\\xAA","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,128 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,129 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"vvv\\xAB","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,130 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,131 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"www\\xAC","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,132 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,133 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"xxx\\xAD","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,134 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,135 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"yyy\\xAE","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,135 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,136 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"\\xAF","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,137 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,138 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"bbb\\xB0","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,139 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,140 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ccc\\xB1","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,141 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,142 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ddd\\xB2","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,143 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,148 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"eee\\xB3","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,150 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,151 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"fff\\xB4","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,152 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,153 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ggg\\xB5","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,154 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,155 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"hhh\\xB6","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,156 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,157 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"iii\\xB7","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,157 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,159 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"jjj\\xB8","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,159 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,160 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"kkk\\xB9","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,161 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,162 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"lll\\xBA","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,163 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,164 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"mmm\\xBB","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,164 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,165 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"nnn\\xBC","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,166 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,167 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ooo\\xBD","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,168 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,168 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ppp\\xBE","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,169 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,170 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"qqq\\xBF","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,171 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,171 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"rrr\\xC0","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,172 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,173 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"sss\\xC1","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,174 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,174 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ttt\\xC2","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,175 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,176 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"uuu\\xC3","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,177 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,177 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"vvv\\xC4","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,178 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,179 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"www\\xC5","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,179 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,180 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"xxx\\xC6","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,181 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,182 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"yyy\\xC7","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,183 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,183 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"\\xC8","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,184 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,185 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"bbb\\xC9","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,185 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,186 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ccc\\xCA","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,187 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,187 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ddd\\xCB","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,188 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,189 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"eee\\xCC","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,189 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,190 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"fff\\xCD","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,191 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,191 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ggg\\xCE","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,192 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,193 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"hhh\\xCF","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,193 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,194 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"iii\\xD0","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,195 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,196 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"jjj\\xD1","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,196 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,197 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"kkk\\xD2","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,198 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,198 INFO [master/asf911:0] balancer.BaseLoadBalancer(1497): Reassigned 1 regions. 1 retained the pre-restart assignment. 2018-06-21 23:23:40,198 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"lll\\xD3","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,199 INFO [PEWorker-7] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:40,199 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state CLOSING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,317 INFO [PEWorker-7] assignment.RegionTransitionProcedure(241): Dispatch pid=67, ppid=64, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:40,317 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"mmm\\xD4","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,318 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,319 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"nnn\\xD5","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,320 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,321 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ooo\\xD6","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,321 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,322 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ppp\\xD7","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,323 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,324 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"qqq\\xD8","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,325 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,326 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"rrr\\xD9","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,327 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,328 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"sss\\xDA","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,328 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,330 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ttt\\xDB","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,330 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,331 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"uuu\\xDC","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,332 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,333 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"vvv\\xDD","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,334 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,335 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"www\\xDE","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,336 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,337 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"xxx\\xDF","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,338 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,339 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"yyy\\xE0","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,342 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,344 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"\\xE1","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,345 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,346 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"bbb\\xE2","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,347 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,348 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ccc\\xE3","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,349 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,350 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ddd\\xE4","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,351 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,352 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"eee\\xE5","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,353 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,354 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"fff\\xE6","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,354 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,355 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ggg\\xE7","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,356 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,357 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"hhh\\xE8","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,358 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,359 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"iii\\xE9","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,359 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,360 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"jjj\\xEA","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,364 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,364 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"kkk\\xEB","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,370 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,372 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"lll\\xEC","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,373 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,374 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"mmm\\xED","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,374 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,376 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"nnn\\xEE","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,377 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,377 INFO [RS:5;asf911:50612] wal.AbstractFSWAL(424): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C50612%2C1529623419106, suffix=, logDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,50612,1529623419106, archiveDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/oldWALs 2018-06-21 23:23:40,379 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ooo\\xEF","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,380 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,382 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ppp\\xF0","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,388 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,390 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"qqq\\xF1","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,393 DEBUG [RS-EventLoopGroup-11-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:57095,DS-901750e1-a133-4f10-85af-4c0c301de1f4,DISK] 2018-06-21 23:23:40,393 DEBUG [RS-EventLoopGroup-11-4] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:53553,DS-615ebfc4-105b-4470-b106-59b6b4ae84d8,DISK] 2018-06-21 23:23:40,394 DEBUG [RS-EventLoopGroup-11-5] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33402,DS-189d75fd-2cba-4781-8f7e-7f1fecbda947,DISK] 2018-06-21 23:23:40,394 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,396 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"rrr\\xF2","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,397 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,398 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"sss\\xF3","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,399 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,400 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ttt\\xF4","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,400 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,402 INFO [RS:5;asf911:50612] wal.AbstractFSWAL(679): New WAL /user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,50612,1529623419106/asf911.gq1.ygridcore.net%2C50612%2C1529623419106.1529623420378 2018-06-21 23:23:40,402 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"uuu\\xF5","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,402 DEBUG [RS:5;asf911:50612] wal.AbstractFSWAL(769): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:57095,DS-901750e1-a133-4f10-85af-4c0c301de1f4,DISK], DatanodeInfoWithStorage[127.0.0.1:53553,DS-615ebfc4-105b-4470-b106-59b6b4ae84d8,DISK], DatanodeInfoWithStorage[127.0.0.1:33402,DS-189d75fd-2cba-4781-8f7e-7f1fecbda947,DISK]] 2018-06-21 23:23:40,403 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,404 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"vvv\\xF6","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,404 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,405 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"www\\xF7","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,406 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,410 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"xxx\\xF8","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,412 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,413 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"yyy\\xF9","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,413 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,415 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"\\xFA","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,416 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,418 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"bbb\\xFB","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,419 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,421 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ccc\\xFC","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,421 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,422 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ddd\\xFD","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,425 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,426 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"eee\\xFE","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,427 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,428 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"fff\\xFF","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,429 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,430 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ggg\\x00","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,431 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,432 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"hhh\\x01","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,432 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,433 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"\\x00","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,434 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,435 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"bbb\\x01","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,436 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,437 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ccc\\x02","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,437 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,438 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ddd\\x03","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,439 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,440 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"eee\\x04","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,441 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,442 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"fff\\x05","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,442 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,443 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ggg\\x06","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,444 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,445 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"hhh\\x07","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,446 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,447 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"iii\\x08","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,447 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,448 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"jjj\\x09","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,449 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,450 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"kkk\\x0A","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,451 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,452 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"lll\\x0B","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,452 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,453 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"mmm\\x0C","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,454 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,455 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"nnn\\x0D","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,456 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,457 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ooo\\x0E","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,458 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,459 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ppp\\x0F","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,459 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,460 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"qqq\\x10","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,461 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,462 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"rrr\\x11","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,463 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,464 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"sss\\x12","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,465 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,466 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ttt\\x13","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,466 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,467 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(119): Fallback to compat rpc execution for serverName=asf911.gq1.ygridcore.net,55146,1529623388534 version=0 2018-06-21 23:23:40,467 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"uuu\\x14","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,468 INFO [RSProcedureDispatcher-pool3-t39] regionserver.RSRpcServices(1980): Open hbase:meta,,1.1588230740 2018-06-21 23:23:40,468 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,469 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(7094): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2018-06-21 23:23:40,469 DEBUG [RS_OPEN_META-master/asf911:0-0] coprocessor.CoprocessorHost(200): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2018-06-21 23:23:40,469 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(8072): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2018-06-21 23:23:40,469 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"vvv\\x15","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,469 INFO [RS_OPEN_META-master/asf911:0-0] regionserver.RegionCoprocessorHost(394): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2018-06-21 23:23:40,470 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table meta 1588230740 2018-06-21 23:23:40,470 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(827): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:40,470 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(7134): checking encryption for 1588230740 2018-06-21 23:23:40,470 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(7139): checking classloading for 1588230740 2018-06-21 23:23:40,470 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,471 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"www\\x16","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,472 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,473 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"xxx\\x17","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,474 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,474 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/info 2018-06-21 23:23:40,475 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/info 2018-06-21 23:23:40,476 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"yyy\\x18","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,476 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=25, currentSize=810.51 KB, freeSize=994.81 MB, maxSize=995.60 MB, heapSize=810.51 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:40,476 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:40,477 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,478 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"\\x19","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,479 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,480 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"bbb\\x1A","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,481 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,482 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ccc\\x1B","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,483 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,484 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ddd\\x1C","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,485 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,486 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"eee\\x1D","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,487 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,488 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(581): loaded hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/info/5284050d0da44000946b8468c73dd88e 2018-06-21 23:23:40,488 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"fff\\x1E","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,489 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,490 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ggg\\x1F","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,491 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,492 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"hhh ","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,493 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,494 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"iii!","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,495 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(581): loaded hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/info/76d2ae477a214062946362815649f5ab 2018-06-21 23:23:40,495 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:40,495 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,496 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"jjj\"","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,497 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/rep_barrier 2018-06-21 23:23:40,497 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/rep_barrier 2018-06-21 23:23:40,497 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,497 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=25, currentSize=810.51 KB, freeSize=994.81 MB, maxSize=995.60 MB, heapSize=810.51 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:40,498 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:40,498 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"kkk#","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,499 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:40,500 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,501 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/table 2018-06-21 23:23:40,501 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/table 2018-06-21 23:23:40,501 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=25, currentSize=810.51 KB, freeSize=994.81 MB, maxSize=995.60 MB, heapSize=810.51 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:40,501 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"lll$","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,502 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:40,502 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,503 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"mmm%","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,504 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,506 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"nnn&","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,506 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,510 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ooo'","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,511 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,512 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ppp(","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,513 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,514 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"qqq)","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,515 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(581): loaded hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/table/cac52b561444446c972f1285f18c4aa4 2018-06-21 23:23:40,515 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:40,516 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(923): replaying wal for 1588230740 2018-06-21 23:23:40,516 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,517 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"rrr*","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,518 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740 2018-06-21 23:23:40,518 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(931): stopping wal replay for 1588230740 2018-06-21 23:23:40,518 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(943): Cleaning up temporary data for 1588230740 2018-06-21 23:23:40,519 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,519 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(954): Cleaning up detritus for 1588230740 2018-06-21 23:23:40,520 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"sss+","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,521 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7M)) instead. 2018-06-21 23:23:40,522 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(976): writing seq id for 1588230740 2018-06-21 23:23:40,523 INFO [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(980): Opened 1588230740; next sequenceid=114 2018-06-21 23:23:40,523 DEBUG [RS_OPEN_META-master/asf911:0-0] regionserver.HRegion(987): Running coprocessor post-open hooks for 1588230740 2018-06-21 23:23:40,523 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,525 ERROR [Time-limited test] client.AsyncRequestFutureImpl(492): Cannot get replica 0 location for {"totalColumns":1,"row":"ttt,","families":{"family":[{"qualifier":"qual","vlen":5,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-06-21 23:23:40,525 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2187): Post open deploy tasks for hbase:meta,,1.1588230740 2018-06-21 23:23:40,525 DEBUG [PostOpenDeployTasks:1588230740] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=114, pid=67, ppid=64, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:40,525 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2211): Finished post open deploy task for hbase:meta,,1.1588230740 2018-06-21 23:23:40,526 DEBUG [PEWorker-2] assignment.RegionTransitionProcedure(354): Finishing pid=67, ppid=64, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:40,526 DEBUG [RS_OPEN_META-master/asf911:0-0] handler.OpenRegionHandler(128): Opened hbase:meta,,1.1588230740 on asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:40,526 INFO [PEWorker-2] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:40,529 DEBUG [Time-limited test] client.ConnectionImplementation(901): locateRegionInMeta parentTable='hbase:meta', attempt=0 of 16 failed; retrying after sleep of 16 org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location for replica 0 at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:333) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153) at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191) at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:834) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:737) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.findAllLocationsOrFail(AsyncRequestFutureImpl.java:505) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncRequestFutureImpl.java:422) at org.apache.hadoop.hbase.client.AsyncProcess.submitAll(AsyncProcess.java:391) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:226) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:452) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testActiveThreadsCount(TestMultiParallel.java:189) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Meta region is in state OPENING at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$null$1(ZKAsyncRegistry.java:154) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:77) at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:171) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:329) ... 1 more 2018-06-21 23:23:40,637 INFO [RS-EventLoopGroup-1-12] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-06-21 23:23:40,638 INFO [RS-EventLoopGroup-1-12] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 41212 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:40,647 INFO [PEWorker-2] procedure2.ProcedureExecutor(1635): Finished subprocedure(s) of pid=64, state=RUNNABLE; MoveRegionProcedure hri=1588230740, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=; resume parent processing. 2018-06-21 23:23:40,647 INFO [PEWorker-2] procedure2.ProcedureExecutor(1266): Finished pid=67, ppid=64, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 in 536msec 2018-06-21 23:23:40,704 INFO [PEWorker-1] procedure2.ProcedureExecutor(1266): Finished pid=64, state=SUCCESS; MoveRegionProcedure hri=1588230740, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination= in 1.3530sec 2018-06-21 23:23:40,707 INFO [PEWorker-12] procedure.MasterProcedureScheduler(696): pid=65, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=1588230740, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,34033,1529623407773 checking lock on 1588230740 2018-06-21 23:23:40,707 INFO [PEWorker-12] procedure2.ProcedureExecutor(1516): Initialized subprocedures=[{pid=68, ppid=65, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=hbase:meta, region=1588230740, server=asf911.gq1.ygridcore.net,55146,1529623388534}] 2018-06-21 23:23:40,764 INFO [PEWorker-16] procedure.MasterProcedureScheduler(696): pid=68, ppid=65, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=hbase:meta, region=1588230740, server=asf911.gq1.ygridcore.net,55146,1529623388534 checking lock on 1588230740 2018-06-21 23:23:40,764 INFO [PEWorker-16] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:40,775 INFO [PEWorker-16] assignment.RegionTransitionProcedure(241): Dispatch pid=68, ppid=65, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=hbase:meta, region=1588230740, server=asf911.gq1.ygridcore.net,55146,1529623388534; rit=CLOSING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:40,776 INFO [RS-EventLoopGroup-7-18] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-06-21 23:23:40,776 INFO [RS-EventLoopGroup-7-18] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 43523 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:40,777 INFO [RS-EventLoopGroup-10-17] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-06-21 23:23:40,778 INFO [RS-EventLoopGroup-10-17] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 52401 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:40,778 INFO [RS-EventLoopGroup-5-5] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-06-21 23:23:40,778 INFO [RS-EventLoopGroup-5-5] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 59850 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:40,778 INFO [RS-EventLoopGroup-6-5] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-06-21 23:23:40,779 INFO [RS-EventLoopGroup-6-5] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 56991 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:40,838 INFO [Time-limited test] zookeeper.ReadOnlyZKClient(350): Close zookeeper connection 0x60e8eedd to localhost:58989 2018-06-21 23:23:40,838 DEBUG [Time-limited test] ipc.AbstractRpcClient(483): Stopping rpc client 2018-06-21 23:23:40,926 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(119): Fallback to compat rpc execution for serverName=asf911.gq1.ygridcore.net,55146,1529623388534 version=0 2018-06-21 23:23:40,927 INFO [RSProcedureDispatcher-pool3-t41] regionserver.RSRpcServices(1604): Close 1588230740, moving to asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:40,937 DEBUG [RS_CLOSE_META-master/asf911:0-0] regionserver.HRegion(1520): Closing 1588230740, disabling compactions & flushes 2018-06-21 23:23:40,938 DEBUG [RS_CLOSE_META-master/asf911:0-0] regionserver.HRegion(1560): Updates disabled for region hbase:meta,,1.1588230740 2018-06-21 23:23:40,946 INFO [Time-limited test] hbase.ResourceChecker(172): after: client.TestMultiParallel#testActiveThreadsCount Thread=708 (was 668) - Thread LEAK? -, OpenFileDescriptor=2593 (was 2456) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=689 (was 689), ProcessCount=272 (was 271) - ProcessCount LEAK? -, AvailableMemoryMB=13227 (was 13350) 2018-06-21 23:23:40,947 WARN [Time-limited test] hbase.ResourceChecker(135): Thread=708 is superior to 500 2018-06-21 23:23:40,947 WARN [Time-limited test] hbase.ResourceChecker(135): OpenFileDescriptor=2593 is superior to 1024 2018-06-21 23:23:40,975 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@69ed135] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:53553 to delete [blk_1073741889_1065, blk_1073741890_1066, blk_1073741891_1067, blk_1073741882_1058] 2018-06-21 23:23:40,975 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@69ed135] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:33402 to delete [blk_1073741888_1064, blk_1073741889_1065, blk_1073741892_1068, blk_1073741887_1063] 2018-06-21 23:23:40,984 DEBUG [RS_CLOSE_META-master/asf911:0-0] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/recovered.edits/116.seqid, newMaxSeqId=116, maxSeqId=113 2018-06-21 23:23:40,985 DEBUG [RS_CLOSE_META-master/asf911:0-0] coprocessor.CoprocessorHost(288): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-06-21 23:23:40,985 INFO [RS_CLOSE_META-master/asf911:0-0] regionserver.HRegion(1674): Closed hbase:meta,,1.1588230740 2018-06-21 23:23:40,985 DEBUG [RS_CLOSE_META-master/asf911:0-0] assignment.RegionTransitionProcedure(264): Received report CLOSED seqId=-1, pid=68, ppid=65, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=hbase:meta, region=1588230740, server=asf911.gq1.ygridcore.net,55146,1529623388534; rit=CLOSING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:40,986 DEBUG [RS_CLOSE_META-master/asf911:0-0] handler.CloseRegionHandler(124): Closed hbase:meta,,1.1588230740 2018-06-21 23:23:40,986 DEBUG [PEWorker-6] assignment.RegionTransitionProcedure(354): Finishing pid=68, ppid=65, state=RUNNABLE:REGION_TRANSITION_FINISH; UnassignProcedure table=hbase:meta, region=1588230740, server=asf911.gq1.ygridcore.net,55146,1529623388534; rit=CLOSING, location=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:40,986 WARN [PEWorker-6] zookeeper.MetaTableLocator(449): Tried to set null ServerName in hbase:meta; skipping -- ServerName required 2018-06-21 23:23:41,005 INFO [Time-limited test] hbase.ResourceChecker(148): before: client.TestMultiParallel#testBatchWithManyColsInOneRowGetAndPut Thread=710, OpenFileDescriptor=2583, MaxFileDescriptor=60000, SystemLoadAverage=689, ProcessCount=271, AvailableMemoryMB=13217 2018-06-21 23:23:41,005 WARN [Time-limited test] hbase.ResourceChecker(135): Thread=710 is superior to 500 2018-06-21 23:23:41,005 WARN [Time-limited test] hbase.ResourceChecker(135): OpenFileDescriptor=2583 is superior to 1024 2018-06-21 23:23:41,005 INFO [Time-limited test] client.TestMultiParallel(117): before 2018-06-21 23:23:41,006 INFO [Time-limited test] hbase.LocalHBaseCluster(270): Not alive RS:1;asf911:40767 2018-06-21 23:23:41,006 INFO [Time-limited test] client.TestMultiParallel(133): before done 2018-06-21 23:23:41,006 INFO [Time-limited test] client.TestMultiParallel(453): test=testBatchWithManyColsInOneRowGetAndPut 2018-06-21 23:23:41,106 INFO [Time-limited test] hbase.ResourceChecker(172): after: client.TestMultiParallel#testBatchWithManyColsInOneRowGetAndPut Thread=709 (was 710), OpenFileDescriptor=2579 (was 2583), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=689 (was 689), ProcessCount=271 (was 271), AvailableMemoryMB=13214 (was 13217) 2018-06-21 23:23:41,106 WARN [Time-limited test] hbase.ResourceChecker(135): Thread=709 is superior to 500 2018-06-21 23:23:41,106 WARN [Time-limited test] hbase.ResourceChecker(135): OpenFileDescriptor=2579 is superior to 1024 2018-06-21 23:23:41,139 INFO [PEWorker-6] procedure2.ProcedureExecutor(1635): Finished subprocedure(s) of pid=65, state=RUNNABLE:MOVE_REGION_ASSIGN; MoveRegionProcedure hri=1588230740, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,34033,1529623407773; resume parent processing. 2018-06-21 23:23:41,139 INFO [PEWorker-11] procedure2.ProcedureExecutor(1516): Initialized subprocedures=[{pid=69, ppid=65, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740, target=asf911.gq1.ygridcore.net,34033,1529623407773}] 2018-06-21 23:23:41,139 INFO [PEWorker-6] procedure2.ProcedureExecutor(1266): Finished pid=68, ppid=65, state=SUCCESS; UnassignProcedure table=hbase:meta, region=1588230740, server=asf911.gq1.ygridcore.net,55146,1529623388534 in 279msec 2018-06-21 23:23:41,159 INFO [Time-limited test] hbase.ResourceChecker(148): before: client.TestMultiParallel#testHTableDeleteWithList Thread=709, OpenFileDescriptor=2579, MaxFileDescriptor=60000, SystemLoadAverage=689, ProcessCount=271, AvailableMemoryMB=13207 2018-06-21 23:23:41,159 WARN [Time-limited test] hbase.ResourceChecker(135): Thread=709 is superior to 500 2018-06-21 23:23:41,159 WARN [Time-limited test] hbase.ResourceChecker(135): OpenFileDescriptor=2579 is superior to 1024 2018-06-21 23:23:41,160 INFO [Time-limited test] client.TestMultiParallel(117): before 2018-06-21 23:23:41,160 INFO [Time-limited test] hbase.LocalHBaseCluster(270): Not alive RS:1;asf911:40767 2018-06-21 23:23:41,160 INFO [Time-limited test] client.TestMultiParallel(133): before done 2018-06-21 23:23:41,160 INFO [Time-limited test] client.TestMultiParallel(423): test=testHTableDeleteWithList 2018-06-21 23:23:41,224 INFO [PEWorker-9] procedure.MasterProcedureScheduler(696): pid=69, ppid=65, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740, target=asf911.gq1.ygridcore.net,34033,1529623407773 checking lock on 1588230740 2018-06-21 23:23:41,224 INFO [PEWorker-9] assignment.AssignProcedure(218): Starting pid=69, ppid=65, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740, target=asf911.gq1.ygridcore.net,34033,1529623407773; rit=OFFLINE, location=asf911.gq1.ygridcore.net,34033,1529623407773; forceNewPlan=false, retain=false 2018-06-21 23:23:41,374 INFO [master/asf911:0] balancer.BaseLoadBalancer(1497): Reassigned 1 regions. 1 retained the pre-restart assignment. 2018-06-21 23:23:41,375 INFO [PEWorker-4] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:41,400 INFO [PEWorker-4] assignment.RegionTransitionProcedure(241): Dispatch pid=69, ppid=65, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740, target=asf911.gq1.ygridcore.net,34033,1529623407773; rit=OPENING, location=asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:41,554 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=34033] regionserver.RSRpcServices(1980): Open hbase:meta,,1.1588230740 2018-06-21 23:23:41,555 INFO [RS_OPEN_META-regionserver/asf911:0-0] wal.WALFactory(137): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-06-21 23:23:41,561 INFO [RS_OPEN_META-regionserver/asf911:0-0] wal.AbstractFSWAL(424): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=asf911.gq1.ygridcore.net%2C34033%2C1529623407773.meta, suffix=.meta, logDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,34033,1529623407773, archiveDir=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/oldWALs 2018-06-21 23:23:41,580 DEBUG [RS-EventLoopGroup-11-9] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:57095,DS-901750e1-a133-4f10-85af-4c0c301de1f4,DISK] 2018-06-21 23:23:41,581 DEBUG [RS-EventLoopGroup-11-10] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:54793,DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e,DISK] 2018-06-21 23:23:41,582 DEBUG [RS-EventLoopGroup-11-11] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:33402,DS-189d75fd-2cba-4781-8f7e-7f1fecbda947,DISK] 2018-06-21 23:23:41,604 INFO [RS_OPEN_META-regionserver/asf911:0-0] wal.AbstractFSWAL(679): New WAL /user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,34033,1529623407773/asf911.gq1.ygridcore.net%2C34033%2C1529623407773.meta.1529623421561.meta 2018-06-21 23:23:41,604 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] wal.AbstractFSWAL(769): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:57095,DS-901750e1-a133-4f10-85af-4c0c301de1f4,DISK], DatanodeInfoWithStorage[127.0.0.1:54793,DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e,DISK], DatanodeInfoWithStorage[127.0.0.1:33402,DS-189d75fd-2cba-4781-8f7e-7f1fecbda947,DISK]] 2018-06-21 23:23:41,605 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(7094): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2018-06-21 23:23:41,605 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] coprocessor.CoprocessorHost(200): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2018-06-21 23:23:41,605 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(8072): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2018-06-21 23:23:41,605 INFO [RS_OPEN_META-regionserver/asf911:0-0] regionserver.RegionCoprocessorHost(394): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2018-06-21 23:23:41,606 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table meta 1588230740 2018-06-21 23:23:41,606 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(827): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-06-21 23:23:41,606 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(7134): checking encryption for 1588230740 2018-06-21 23:23:41,606 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(7139): checking classloading for 1588230740 2018-06-21 23:23:41,610 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/info 2018-06-21 23:23:41,610 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/info 2018-06-21 23:23:41,611 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=27, currentSize=819.33 KB, freeSize=994.80 MB, maxSize=995.60 MB, heapSize=819.33 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:41,612 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:41,629 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(581): loaded hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/info/5284050d0da44000946b8468c73dd88e 2018-06-21 23:23:41,634 INFO [Time-limited test] hbase.ResourceChecker(172): after: client.TestMultiParallel#testHTableDeleteWithList Thread=729 (was 709) - Thread LEAK? -, OpenFileDescriptor=2591 (was 2579) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=689 (was 689), ProcessCount=271 (was 271), AvailableMemoryMB=13109 (was 13207) 2018-06-21 23:23:41,634 WARN [Time-limited test] hbase.ResourceChecker(135): Thread=729 is superior to 500 2018-06-21 23:23:41,634 WARN [Time-limited test] hbase.ResourceChecker(135): OpenFileDescriptor=2591 is superior to 1024 2018-06-21 23:23:41,651 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(581): loaded hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/info/76d2ae477a214062946362815649f5ab 2018-06-21 23:23:41,652 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:41,654 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/rep_barrier 2018-06-21 23:23:41,654 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/rep_barrier 2018-06-21 23:23:41,654 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=27, currentSize=819.33 KB, freeSize=994.80 MB, maxSize=995.60 MB, heapSize=819.33 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:41,655 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:41,656 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:41,658 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/table 2018-06-21 23:23:41,658 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(565): Set storagePolicy=HOT for path=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/table 2018-06-21 23:23:41,658 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=27, currentSize=819.33 KB, freeSize=994.80 MB, maxSize=995.60 MB, heapSize=819.33 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-06-21 23:23:41,659 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-06-21 23:23:41,667 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(581): loaded hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/table/cac52b561444446c972f1285f18c4aa4 2018-06-21 23:23:41,668 INFO [StoreOpener-1588230740-1] regionserver.HStore(327): Store=table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50 2018-06-21 23:23:41,668 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(923): replaying wal for 1588230740 2018-06-21 23:23:41,670 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(4475): Found 0 recovered edits file(s) under hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740 2018-06-21 23:23:41,670 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(931): stopping wal replay for 1588230740 2018-06-21 23:23:41,670 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(943): Cleaning up temporary data for 1588230740 2018-06-21 23:23:41,671 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(954): Cleaning up detritus for 1588230740 2018-06-21 23:23:41,672 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7M)) instead. 2018-06-21 23:23:41,673 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(976): writing seq id for 1588230740 2018-06-21 23:23:41,674 INFO [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(980): Opened 1588230740; next sequenceid=117 2018-06-21 23:23:41,674 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] regionserver.HRegion(987): Running coprocessor post-open hooks for 1588230740 2018-06-21 23:23:41,676 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2187): Post open deploy tasks for hbase:meta,,1.1588230740 2018-06-21 23:23:41,677 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=55146] assignment.RegionTransitionProcedure(264): Received report OPENED seqId=117, pid=69, ppid=65, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740, target=asf911.gq1.ygridcore.net,34033,1529623407773; rit=OPENING, location=asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:41,677 DEBUG [PEWorker-14] assignment.RegionTransitionProcedure(354): Finishing pid=69, ppid=65, state=RUNNABLE:REGION_TRANSITION_FINISH; AssignProcedure table=hbase:meta, region=1588230740, target=asf911.gq1.ygridcore.net,34033,1529623407773; rit=OPENING, location=asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:41,678 INFO [PEWorker-14] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:41,678 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2211): Finished post open deploy task for hbase:meta,,1.1588230740 2018-06-21 23:23:41,678 DEBUG [RS_OPEN_META-regionserver/asf911:0-0] handler.OpenRegionHandler(128): Opened hbase:meta,,1.1588230740 on asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:41,680 INFO [Time-limited test] hbase.ResourceChecker(148): before: client.TestMultiParallel#testBadFam Thread=734, OpenFileDescriptor=2591, MaxFileDescriptor=60000, SystemLoadAverage=689, ProcessCount=271, AvailableMemoryMB=13107 2018-06-21 23:23:41,681 WARN [Time-limited test] hbase.ResourceChecker(135): Thread=734 is superior to 500 2018-06-21 23:23:41,681 WARN [Time-limited test] hbase.ResourceChecker(135): OpenFileDescriptor=2591 is superior to 1024 2018-06-21 23:23:41,681 INFO [Time-limited test] client.TestMultiParallel(117): before 2018-06-21 23:23:41,681 INFO [Time-limited test] hbase.LocalHBaseCluster(270): Not alive RS:1;asf911:40767 2018-06-21 23:23:41,681 INFO [Time-limited test] client.TestMultiParallel(133): before done 2018-06-21 23:23:41,681 INFO [Time-limited test] client.TestMultiParallel(246): test=testBadFam 2018-06-21 23:23:41,683 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=58235] regionserver.HRegion$BatchOperation(3159): No such column family in batch mutation. org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family bad_family does not exist in region multi_test_table,qqq,1529623399386.7480a111a1f600aa34b97a77e47640bc. in table 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at org.apache.hadoop.hbase.regionserver.HRegion.checkFamily(HRegion.java:7999) at org.apache.hadoop.hbase.regionserver.HRegion.checkFamilies(HRegion.java:4386) at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.checkAndPreparePut(HRegion.java:3485) at org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.checkAndPrepareMutation(HRegion.java:3135) at org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.checkAndPrepareMutation(HRegion.java:3145) at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation$1.visit(HRegion.java:3511) at org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.visitBatchOperations(HRegion.java:3037) at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.checkAndPrepare(HRegion.java:3491) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3930) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3864) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1029) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:961) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:924) at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2688) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) 2018-06-21 23:23:41,689 WARN [hconnection-0xe1c5692-shared-pool15-t53] client.AsyncRequestFutureImpl(790): id=7, table=multi_test_table, attempt=1/16, failed=1ops, last exception=org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family bad_family does not exist in region multi_test_table,qqq,1529623399386.7480a111a1f600aa34b97a77e47640bc. in table 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1038) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:961) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:924) at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2688) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) on asf911.gq1.ygridcore.net,58235,1529623390382, tracking started Thu Jun 21 23:23:41 UTC 2018; not retrying 1 - final failure 2018-06-21 23:23:41,689 DEBUG [Time-limited test] client.TestMultiParallel(264): org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family bad_family does not exist in region multi_test_table,qqq,1529623399386.7480a111a1f600aa34b97a77e47640bc. in table 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1038) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:961) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:924) at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2688) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) : 1 time, servers with issues: asf911.gq1.ygridcore.net,58235,1529623390382 org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family bad_family does not exist in region multi_test_table,qqq,1529623399386.7480a111a1f600aa34b97a77e47640bc. in table 'multi_test_table', {NAME => 'family', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1038) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:961) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:924) at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2688) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42014) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) : 1 time, servers with issues: asf911.gq1.ygridcore.net,58235,1529623390382 at org.apache.hadoop.hbase.client.BatchErrors.makeException(BatchErrors.java:54) at org.apache.hadoop.hbase.client.AsyncRequestFutureImpl.getErrors(AsyncRequestFutureImpl.java:1225) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:455) at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:438) at org.apache.hadoop.hbase.client.TestMultiParallel.testBadFam(TestMultiParallel.java:261) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) 2018-06-21 23:23:41,734 INFO [Time-limited test] hbase.ResourceChecker(172): after: client.TestMultiParallel#testBadFam Thread=733 (was 734), OpenFileDescriptor=2591 (was 2591), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=689 (was 689), ProcessCount=271 (was 271), AvailableMemoryMB=13107 (was 13107) 2018-06-21 23:23:41,734 WARN [Time-limited test] hbase.ResourceChecker(135): Thread=733 is superior to 500 2018-06-21 23:23:41,734 WARN [Time-limited test] hbase.ResourceChecker(135): OpenFileDescriptor=2591 is superior to 1024 2018-06-21 23:23:41,778 INFO [Time-limited test] hbase.ResourceChecker(148): before: client.TestMultiParallel#testFlushCommitsNoAbort Thread=733, OpenFileDescriptor=2591, MaxFileDescriptor=60000, SystemLoadAverage=689, ProcessCount=271, AvailableMemoryMB=13106 2018-06-21 23:23:41,779 WARN [Time-limited test] hbase.ResourceChecker(135): Thread=733 is superior to 500 2018-06-21 23:23:41,779 WARN [Time-limited test] hbase.ResourceChecker(135): OpenFileDescriptor=2591 is superior to 1024 2018-06-21 23:23:41,779 INFO [Time-limited test] client.TestMultiParallel(117): before 2018-06-21 23:23:41,779 INFO [Time-limited test] hbase.LocalHBaseCluster(270): Not alive RS:1;asf911:40767 2018-06-21 23:23:41,779 INFO [Time-limited test] client.TestMultiParallel(133): before done 2018-06-21 23:23:41,779 INFO [Time-limited test] client.TestMultiParallel(276): test=testFlushCommitsNoAbort 2018-06-21 23:23:41,779 INFO [Time-limited test] client.TestMultiParallel(299): get new table 2018-06-21 23:23:41,779 INFO [Time-limited test] client.TestMultiParallel(302): constructPutRequests 2018-06-21 23:23:41,782 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=55146] ipc.CallRunner(142): callId: 922 service: ClientService methodName: Scan size: 146 connection: 67.195.81.155:40712 deadline: 1529623481782, exception=org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:41,795 INFO [PEWorker-14] procedure2.ProcedureExecutor(1635): Finished subprocedure(s) of pid=65, state=RUNNABLE; MoveRegionProcedure hri=1588230740, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,34033,1529623407773; resume parent processing. 2018-06-21 23:23:41,796 INFO [PEWorker-14] procedure2.ProcedureExecutor(1266): Finished pid=69, ppid=65, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740, target=asf911.gq1.ygridcore.net,34033,1529623407773 in 544msec 2018-06-21 23:23:41,921 INFO [Time-limited test] client.TestMultiParallel(305): puts 2018-06-21 23:23:41,921 INFO [Time-limited test] hbase.LocalHBaseCluster(270): Not alive RS:1;asf911:40767 2018-06-21 23:23:41,921 INFO [Time-limited test] hbase.LocalHBaseCluster(270): Not alive RS:1;asf911:40767 2018-06-21 23:23:41,922 INFO [Time-limited test] client.TestMultiParallel(327): validating loaded data 2018-06-21 23:23:41,922 INFO [Time-limited test] client.TestMultiParallel(755): Validating data on multi_test_table;hconnection-0xe1c5692 2018-06-21 23:23:41,939 INFO [PEWorker-5] procedure2.ProcedureExecutor(1266): Finished pid=65, state=SUCCESS; MoveRegionProcedure hri=1588230740, source=asf911.gq1.ygridcore.net,55146,1529623388534, destination=asf911.gq1.ygridcore.net,34033,1529623407773 in 2.4260sec 2018-06-21 23:23:41,946 INFO [Time-limited test] client.TestMultiParallel(792): Validating data on multi_test_table;hconnection-0xe1c5692 successfully! 2018-06-21 23:23:41,946 INFO [Time-limited test] hbase.LocalHBaseCluster(270): Not alive RS:1;asf911:40767 2018-06-21 23:23:41,946 INFO [Time-limited test] client.TestMultiParallel(335): Count=1, Alive=asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:41,946 INFO [Time-limited test] client.TestMultiParallel(335): Count=2, Alive=asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:41,946 INFO [Time-limited test] client.TestMultiParallel(335): Count=3, Alive=asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:41,946 INFO [Time-limited test] client.TestMultiParallel(335): Count=4, Alive=asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:41,946 INFO [Time-limited test] client.TestMultiParallel(335): Count=5, Alive=asf911.gq1.ygridcore.net,50612,1529623419106 2018-06-21 23:23:41,946 INFO [Time-limited test] client.TestMultiParallel(337): Count=5 2018-06-21 23:23:41,946 INFO [Time-limited test] client.TestMultiParallel(354): done 2018-06-21 23:23:41,993 INFO [Time-limited test] hbase.ResourceChecker(172): after: client.TestMultiParallel#testFlushCommitsNoAbort Thread=730 (was 733), OpenFileDescriptor=2591 (was 2591), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=689 (was 689), ProcessCount=271 (was 271), AvailableMemoryMB=13107 (was 13106) - AvailableMemoryMB LEAK? - 2018-06-21 23:23:41,993 WARN [Time-limited test] hbase.ResourceChecker(135): Thread=730 is superior to 500 2018-06-21 23:23:41,994 WARN [Time-limited test] hbase.ResourceChecker(135): OpenFileDescriptor=2591 is superior to 1024 2018-06-21 23:23:42,041 INFO [Time-limited test] hbase.ResourceChecker(148): before: client.TestMultiParallel#testBatchWithDelete Thread=727, OpenFileDescriptor=2591, MaxFileDescriptor=60000, SystemLoadAverage=689, ProcessCount=271, AvailableMemoryMB=13108 2018-06-21 23:23:42,041 WARN [Time-limited test] hbase.ResourceChecker(135): Thread=727 is superior to 500 2018-06-21 23:23:42,041 WARN [Time-limited test] hbase.ResourceChecker(135): OpenFileDescriptor=2591 is superior to 1024 2018-06-21 23:23:42,041 INFO [Time-limited test] client.TestMultiParallel(117): before 2018-06-21 23:23:42,042 INFO [Time-limited test] hbase.LocalHBaseCluster(270): Not alive RS:1;asf911:40767 2018-06-21 23:23:42,042 INFO [Time-limited test] client.TestMultiParallel(133): before done 2018-06-21 23:23:42,042 INFO [Time-limited test] client.TestMultiParallel(392): test=testBatchWithDelete 2018-06-21 23:23:42,457 INFO [Time-limited test] hbase.ResourceChecker(172): after: client.TestMultiParallel#testBatchWithDelete Thread=735 (was 727) - Thread LEAK? -, OpenFileDescriptor=2591 (was 2591), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=689 (was 689), ProcessCount=271 (was 271), AvailableMemoryMB=13108 (was 13108) 2018-06-21 23:23:42,458 WARN [Time-limited test] hbase.ResourceChecker(135): Thread=735 is superior to 500 2018-06-21 23:23:42,458 WARN [Time-limited test] hbase.ResourceChecker(135): OpenFileDescriptor=2591 is superior to 1024 2018-06-21 23:23:42,508 INFO [Time-limited test] hbase.ResourceChecker(148): before: client.TestMultiParallel#testNonceCollision Thread=735, OpenFileDescriptor=2591, MaxFileDescriptor=60000, SystemLoadAverage=689, ProcessCount=271, AvailableMemoryMB=13108 2018-06-21 23:23:42,508 WARN [Time-limited test] hbase.ResourceChecker(135): Thread=735 is superior to 500 2018-06-21 23:23:42,508 WARN [Time-limited test] hbase.ResourceChecker(135): OpenFileDescriptor=2591 is superior to 1024 2018-06-21 23:23:42,508 INFO [Time-limited test] client.TestMultiParallel(117): before 2018-06-21 23:23:42,508 INFO [Time-limited test] hbase.LocalHBaseCluster(270): Not alive RS:1;asf911:40767 2018-06-21 23:23:42,508 INFO [Time-limited test] client.TestMultiParallel(133): before done 2018-06-21 23:23:42,508 INFO [Time-limited test] client.TestMultiParallel(527): test=testNonceCollision 2018-06-21 23:23:42,509 INFO [Time-limited test] zookeeper.ReadOnlyZKClient(139): Connect 0x66c2cb03 to localhost:58989 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-06-21 23:23:42,551 DEBUG [Time-limited test] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@180b8704, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-06-21 23:23:42,552 WARN [Time-limited test] client.ConnectionImplementation(333): Nonce generator is being replaced by test code for org.apache.hadoop.hbase.client.TestMultiParallel$2 2018-06-21 23:23:42,561 INFO [RS-EventLoopGroup-10-18] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-06-21 23:23:42,561 INFO [RS-EventLoopGroup-10-18] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 52424 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:42,573 INFO [RS-EventLoopGroup-6-6] ipc.ServerRpcConnection(528): Auth successful for jenkins (auth:SIMPLE) 2018-06-21 23:23:42,573 INFO [RS-EventLoopGroup-6-6] ipc.ServerRpcConnection(558): Connection from 67.195.81.155 port: 57016 with version info: version: "3.0.0-SNAPSHOT" url: "git://asf911.gq1.ygridcore.net/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests" revision: "bc9f9ae080c1d6e8f6ed400883cc2d756bbc3fa2" user: "jenkins" date: "Thu Jun 21 23:19:17 UTC 2018" src_checksum: "cd772cfe76608f6dad5341f36e5e39c8" version_major: 3 version_minor: 0 2018-06-21 23:23:42,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:-5653001525922928385], [state 0, hasWait false, activity 23:23:42.577] 2018-06-21 23:23:42,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:1392491012333869557], [state 2, hasWait false, activity 23:23:42.600] 2018-06-21 23:23:42,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:-5172801255604871334], [state 2, hasWait false, activity 23:23:42.600] 2018-06-21 23:23:42,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56613] regionserver.ServerNonceManager(197): Conflict with running op ended: [6127138109825379235:1392491012333869557], [state 0, hasWait true, activity 23:23:42.606] 2018-06-21 23:23:42,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:2085445760723557683], [state 2, hasWait false, activity 23:23:42.600] 2018-06-21 23:23:42,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:1392491012333869557], [state 0, hasWait true, activity 23:23:42.606] 2018-06-21 23:23:42,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=56613] regionserver.ServerNonceManager(197): Conflict with running op ended: [6127138109825379235:-5172801255604871334], [state 0, hasWait true, activity 23:23:42.612] 2018-06-21 23:23:42,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:-5172801255604871334], [state 0, hasWait true, activity 23:23:42.612] 2018-06-21 23:23:42,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:-5514163463636425755], [state 2, hasWait false, activity 23:23:42.609] 2018-06-21 23:23:42,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=56613] regionserver.ServerNonceManager(197): Conflict with running op ended: [6127138109825379235:2085445760723557683], [state 0, hasWait true, activity 23:23:42.614] 2018-06-21 23:23:42,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:-6846978418798696234], [state 2, hasWait false, activity 23:23:42.613] 2018-06-21 23:23:42,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:2085445760723557683], [state 0, hasWait true, activity 23:23:42.614] 2018-06-21 23:23:42,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=56613] regionserver.ServerNonceManager(197): Conflict with running op ended: [6127138109825379235:-5514163463636425755], [state 0, hasWait true, activity 23:23:42.615] 2018-06-21 23:23:42,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:-5514163463636425755], [state 0, hasWait true, activity 23:23:42.615] 2018-06-21 23:23:42,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:4751913854993180886], [state 2, hasWait false, activity 23:23:42.615] 2018-06-21 23:23:42,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=56613] regionserver.ServerNonceManager(197): Conflict with running op ended: [6127138109825379235:-6846978418798696234], [state 0, hasWait true, activity 23:23:42.616] 2018-06-21 23:23:42,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:3535383745292037780], [state 2, hasWait false, activity 23:23:42.616] 2018-06-21 23:23:42,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:-6846978418798696234], [state 0, hasWait true, activity 23:23:42.616] 2018-06-21 23:23:42,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56613] regionserver.ServerNonceManager(197): Conflict with running op ended: [6127138109825379235:4751913854993180886], [state 0, hasWait true, activity 23:23:42.617] 2018-06-21 23:23:42,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:4751913854993180886], [state 0, hasWait true, activity 23:23:42.617] 2018-06-21 23:23:42,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:-18608667897092230], [state 2, hasWait false, activity 23:23:42.617] 2018-06-21 23:23:42,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=56613] regionserver.ServerNonceManager(197): Conflict with running op ended: [6127138109825379235:3535383745292037780], [state 0, hasWait true, activity 23:23:42.648] 2018-06-21 23:23:42,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:3535383745292037780], [state 0, hasWait true, activity 23:23:42.648] 2018-06-21 23:23:42,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:982727902823431061], [state 2, hasWait false, activity 23:23:42.618] 2018-06-21 23:23:42,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56613] regionserver.ServerNonceManager(197): Conflict with running op ended: [6127138109825379235:-18608667897092230], [state 0, hasWait true, activity 23:23:42.649] 2018-06-21 23:23:42,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=56613] regionserver.ServerNonceManager(197): Conflict with running op ended: [6127138109825379235:982727902823431061], [state 0, hasWait true, activity 23:23:42.651] 2018-06-21 23:23:42,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:982727902823431061], [state 0, hasWait true, activity 23:23:42.651] 2018-06-21 23:23:42,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:-8882574119529918084], [state 2, hasWait false, activity 23:23:42.649] 2018-06-21 23:23:42,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:-18608667897092230], [state 0, hasWait true, activity 23:23:42.649] 2018-06-21 23:23:42,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:6205858144944723286], [state 2, hasWait false, activity 23:23:42.650] 2018-06-21 23:23:42,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=56613] regionserver.ServerNonceManager(197): Conflict with running op ended: [6127138109825379235:-8882574119529918084], [state 0, hasWait true, activity 23:23:42.655] 2018-06-21 23:23:42,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:-8882574119529918084], [state 0, hasWait true, activity 23:23:42.655] 2018-06-21 23:23:42,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:271250689431605863], [state 2, hasWait false, activity 23:23:42.652] 2018-06-21 23:23:42,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56613] regionserver.ServerNonceManager(197): Conflict with running op ended: [6127138109825379235:6205858144944723286], [state 0, hasWait true, activity 23:23:42.656] 2018-06-21 23:23:42,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:6205858144944723286], [state 0, hasWait true, activity 23:23:42.656] 2018-06-21 23:23:42,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:-5480461613953711920], [state 2, hasWait false, activity 23:23:42.655] 2018-06-21 23:23:42,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=56613] regionserver.ServerNonceManager(197): Conflict with running op ended: [6127138109825379235:271250689431605863], [state 0, hasWait true, activity 23:23:42.658] 2018-06-21 23:23:42,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:271250689431605863], [state 0, hasWait true, activity 23:23:42.658] 2018-06-21 23:23:42,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:4175508674331059560], [state 2, hasWait false, activity 23:23:42.656] 2018-06-21 23:23:42,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=56613] regionserver.ServerNonceManager(197): Conflict with running op ended: [6127138109825379235:-5480461613953711920], [state 0, hasWait true, activity 23:23:42.659] 2018-06-21 23:23:42,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:-80133373229492264], [state 2, hasWait false, activity 23:23:42.659] 2018-06-21 23:23:42,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:-5480461613953711920], [state 0, hasWait true, activity 23:23:42.659] 2018-06-21 23:23:42,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56613] regionserver.ServerNonceManager(197): Conflict with running op ended: [6127138109825379235:4175508674331059560], [state 0, hasWait true, activity 23:23:42.661] 2018-06-21 23:23:42,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:4175508674331059560], [state 0, hasWait true, activity 23:23:42.661] 2018-06-21 23:23:42,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:-5229609488226450600], [state 2, hasWait false, activity 23:23:42.660] 2018-06-21 23:23:42,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=56613] regionserver.ServerNonceManager(197): Conflict with running op ended: [6127138109825379235:-80133373229492264], [state 0, hasWait true, activity 23:23:42.662] 2018-06-21 23:23:42,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:-80133373229492264], [state 0, hasWait true, activity 23:23:42.662] 2018-06-21 23:23:42,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=56613] regionserver.ServerNonceManager(197): Conflict with running op ended: [6127138109825379235:-5229609488226450600], [state 0, hasWait true, activity 23:23:42.663] 2018-06-21 23:23:42,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:-5229609488226450600], [state 0, hasWait true, activity 23:23:42.663] 2018-06-21 23:23:42,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:3962252991670721631], [state 2, hasWait false, activity 23:23:42.663] 2018-06-21 23:23:42,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:-8892127960145908486], [state 2, hasWait false, activity 23:23:42.662] 2018-06-21 23:23:42,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=56613] regionserver.ServerNonceManager(197): Conflict with running op ended: [6127138109825379235:-8892127960145908486], [state 0, hasWait true, activity 23:23:42.665] 2018-06-21 23:23:42,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:-8892127960145908486], [state 0, hasWait true, activity 23:23:42.665] 2018-06-21 23:23:42,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:6340489611482289536], [state 2, hasWait false, activity 23:23:42.665] 2018-06-21 23:23:42,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:-8877885679990976312], [state 0, hasWait false, activity 23:23:42.664] 2018-06-21 23:23:42,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=56613] regionserver.ServerNonceManager(197): Conflict with running op ended: [6127138109825379235:3962252991670721631], [state 0, hasWait true, activity 23:23:42.674] 2018-06-21 23:23:42,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:3962252991670721631], [state 0, hasWait true, activity 23:23:42.674] 2018-06-21 23:23:42,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=56613] regionserver.ServerNonceManager(197): Conflict with running op ended: [6127138109825379235:6340489611482289536], [state 0, hasWait true, activity 23:23:42.675] 2018-06-21 23:23:42,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=56613] regionserver.ServerNonceManager(162): Conflict detected by nonce: [6127138109825379235:6340489611482289536], [state 0, hasWait true, activity 23:23:42.675] 2018-06-21 23:23:42,678 WARN [Time-limited test] client.ConnectionImplementation(333): Nonce generator is being replaced by test code for org.apache.hadoop.hbase.client.PerClientRandomNonceGenerator 2018-06-21 23:23:42,727 INFO [Time-limited test] hbase.ResourceChecker(172): after: client.TestMultiParallel#testNonceCollision Thread=741 (was 735) - Thread LEAK? -, OpenFileDescriptor=2600 (was 2591) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=642 (was 689), ProcessCount=271 (was 271), AvailableMemoryMB=13105 (was 13108) 2018-06-21 23:23:42,728 WARN [Time-limited test] hbase.ResourceChecker(135): Thread=741 is superior to 500 2018-06-21 23:23:42,728 WARN [Time-limited test] hbase.ResourceChecker(135): OpenFileDescriptor=2600 is superior to 1024 2018-06-21 23:23:42,774 INFO [Time-limited test] hbase.ResourceChecker(148): before: client.TestMultiParallel#testBatchWithMixedActions Thread=741, OpenFileDescriptor=2600, MaxFileDescriptor=60000, SystemLoadAverage=642, ProcessCount=271, AvailableMemoryMB=13103 2018-06-21 23:23:42,775 WARN [Time-limited test] hbase.ResourceChecker(135): Thread=741 is superior to 500 2018-06-21 23:23:42,775 WARN [Time-limited test] hbase.ResourceChecker(135): OpenFileDescriptor=2600 is superior to 1024 2018-06-21 23:23:42,775 INFO [Time-limited test] client.TestMultiParallel(117): before 2018-06-21 23:23:42,775 INFO [Time-limited test] hbase.LocalHBaseCluster(270): Not alive RS:1;asf911:40767 2018-06-21 23:23:42,775 INFO [Time-limited test] client.TestMultiParallel(133): before done 2018-06-21 23:23:42,776 INFO [Time-limited test] client.TestMultiParallel(627): test=testBatchWithMixedActions 2018-06-21 23:23:42,849 INFO [Time-limited test] hbase.ResourceChecker(172): after: client.TestMultiParallel#testBatchWithMixedActions Thread=748 (was 741) - Thread LEAK? -, OpenFileDescriptor=2600 (was 2600), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=642 (was 642), ProcessCount=271 (was 271), AvailableMemoryMB=13104 (was 13103) - AvailableMemoryMB LEAK? - 2018-06-21 23:23:42,850 WARN [Time-limited test] hbase.ResourceChecker(135): Thread=748 is superior to 500 2018-06-21 23:23:42,850 WARN [Time-limited test] hbase.ResourceChecker(135): OpenFileDescriptor=2600 is superior to 1024 2018-06-21 23:23:42,850 INFO [Time-limited test] zookeeper.ReadOnlyZKClient(350): Close zookeeper connection 0x4cb09ea8 to localhost:58989 2018-06-21 23:23:42,850 DEBUG [Time-limited test] ipc.AbstractRpcClient(483): Stopping rpc client 2018-06-21 23:23:42,851 INFO [Time-limited test] hbase.HBaseTestingUtility(1097): Shutting down minicluster 2018-06-21 23:23:42,851 INFO [Time-limited test] client.ConnectionImplementation(1754): Closing master protocol: MasterService 2018-06-21 23:23:42,853 INFO [Time-limited test] zookeeper.ReadOnlyZKClient(350): Close zookeeper connection 0x3275446e to localhost:58989 2018-06-21 23:23:42,853 DEBUG [Time-limited test] ipc.AbstractRpcClient(483): Stopping rpc client 2018-06-21 23:23:42,854 DEBUG [Time-limited test] util.JVMClusterUtil(238): Shutting down HBase Cluster 2018-06-21 23:23:42,854 DEBUG [Time-limited test] coprocessor.CoprocessorHost(288): Stop coprocessor org.apache.hadoop.hbase.client.TestMultiParallel$MyMasterObserver 2018-06-21 23:23:42,854 INFO [Time-limited test] master.ServerManager(916): Cluster shutdown requested of master=asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:42,866 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:50612-0x16424a89dd40011, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-06-21 23:23:42,866 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-06-21 23:23:42,867 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-06-21 23:23:42,866 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-06-21 23:23:42,866 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-06-21 23:23:42,866 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-06-21 23:23:42,867 INFO [Time-limited test] procedure2.ProcedureExecutor(592): Stopping 2018-06-21 23:23:42,867 INFO [Time-limited test] zookeeper.ReadOnlyZKClient(350): Close zookeeper connection 0x0794c832 to localhost:58989 2018-06-21 23:23:42,868 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(357): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-06-21 23:23:42,868 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(357): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-06-21 23:23:42,868 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(357): regionserver:50612-0x16424a89dd40011, quorum=localhost:58989, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-06-21 23:23:42,867 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(357): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-06-21 23:23:42,868 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(357): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-06-21 23:23:42,868 DEBUG [Time-limited test] ipc.AbstractRpcClient(483): Stopping rpc client 2018-06-21 23:23:42,868 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(357): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-06-21 23:23:42,869 INFO [Time-limited test] regionserver.HRegionServer(2148): ***** STOPPING region server 'asf911.gq1.ygridcore.net,52507,1529623390228' ***** 2018-06-21 23:23:42,869 INFO [Time-limited test] regionserver.HRegionServer(2162): STOPPED: Shutdown requested 2018-06-21 23:23:42,869 INFO [Time-limited test] regionserver.HRegionServer(2148): ***** STOPPING region server 'asf911.gq1.ygridcore.net,56613,1529623390297' ***** 2018-06-21 23:23:42,869 INFO [Time-limited test] regionserver.HRegionServer(2162): STOPPED: Shutdown requested 2018-06-21 23:23:42,869 INFO [RS:2;asf911:52507] regionserver.SplitLogWorker(166): Sending interrupt to stop the worker thread 2018-06-21 23:23:42,870 INFO [Time-limited test] regionserver.HRegionServer(2148): ***** STOPPING region server 'asf911.gq1.ygridcore.net,58235,1529623390382' ***** 2018-06-21 23:23:42,870 INFO [Time-limited test] regionserver.HRegionServer(2162): STOPPED: Shutdown requested 2018-06-21 23:23:42,870 INFO [RS:3;asf911:56613] regionserver.SplitLogWorker(166): Sending interrupt to stop the worker thread 2018-06-21 23:23:42,870 INFO [RS:4;asf911:58235] regionserver.SplitLogWorker(166): Sending interrupt to stop the worker thread 2018-06-21 23:23:42,870 INFO [SplitLogWorker-asf911:56613] regionserver.SplitLogWorker(148): SplitLogWorker interrupted. Exiting. 2018-06-21 23:23:42,870 INFO [Time-limited test] regionserver.HRegionServer(2148): ***** STOPPING region server 'asf911.gq1.ygridcore.net,34033,1529623407773' ***** 2018-06-21 23:23:42,870 INFO [Time-limited test] regionserver.HRegionServer(2162): STOPPED: Shutdown requested 2018-06-21 23:23:42,870 INFO [Time-limited test] regionserver.HRegionServer(2148): ***** STOPPING region server 'asf911.gq1.ygridcore.net,50612,1529623419106' ***** 2018-06-21 23:23:42,871 INFO [Time-limited test] regionserver.HRegionServer(2162): STOPPED: Shutdown requested 2018-06-21 23:23:42,871 INFO [RS:5;asf911:50612] regionserver.SplitLogWorker(166): Sending interrupt to stop the worker thread 2018-06-21 23:23:42,870 INFO [SplitLogWorker-asf911:52507] regionserver.SplitLogWorker(148): SplitLogWorker interrupted. Exiting. 2018-06-21 23:23:42,870 INFO [RS:2;asf911:52507] regionserver.HeapMemoryManager(221): Stopping 2018-06-21 23:23:42,871 INFO [SplitLogWorker-asf911:52507] regionserver.SplitLogWorker(157): SplitLogWorker asf911.gq1.ygridcore.net,52507,1529623390228 exiting 2018-06-21 23:23:42,871 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(383): MemStoreFlusher.0 exiting 2018-06-21 23:23:42,871 INFO [SplitLogWorker-asf911:50612] regionserver.SplitLogWorker(148): SplitLogWorker interrupted. Exiting. 2018-06-21 23:23:42,871 INFO [RS:5;asf911:50612] regionserver.HeapMemoryManager(221): Stopping 2018-06-21 23:23:42,871 INFO [RS:5;asf911:34033] regionserver.SplitLogWorker(166): Sending interrupt to stop the worker thread 2018-06-21 23:23:42,870 INFO [SplitLogWorker-asf911:56613] regionserver.SplitLogWorker(157): SplitLogWorker asf911.gq1.ygridcore.net,56613,1529623390297 exiting 2018-06-21 23:23:42,870 INFO [SplitLogWorker-asf911:58235] regionserver.SplitLogWorker(148): SplitLogWorker interrupted. Exiting. 2018-06-21 23:23:42,870 INFO [RS:4;asf911:58235] regionserver.HeapMemoryManager(221): Stopping 2018-06-21 23:23:42,870 INFO [RS:3;asf911:56613] regionserver.HeapMemoryManager(221): Stopping 2018-06-21 23:23:42,874 INFO [RS:4;asf911:58235] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager gracefully. 2018-06-21 23:23:42,874 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(383): MemStoreFlusher.0 exiting 2018-06-21 23:23:42,878 INFO [RS:4;asf911:58235] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2018-06-21 23:23:42,874 INFO [SplitLogWorker-asf911:58235] regionserver.SplitLogWorker(157): SplitLogWorker asf911.gq1.ygridcore.net,58235,1529623390382 exiting 2018-06-21 23:23:42,874 INFO [SplitLogWorker-asf911:34033] regionserver.SplitLogWorker(148): SplitLogWorker interrupted. Exiting. 2018-06-21 23:23:42,874 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(383): MemStoreFlusher.1 exiting 2018-06-21 23:23:42,874 INFO [RS:5;asf911:34033] regionserver.HeapMemoryManager(221): Stopping 2018-06-21 23:23:42,873 INFO [RS:5;asf911:50612] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager gracefully. 2018-06-21 23:23:42,873 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(383): MemStoreFlusher.0 exiting 2018-06-21 23:23:42,873 INFO [SplitLogWorker-asf911:50612] regionserver.SplitLogWorker(157): SplitLogWorker asf911.gq1.ygridcore.net,50612,1529623419106 exiting 2018-06-21 23:23:42,872 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(383): MemStoreFlusher.1 exiting 2018-06-21 23:23:42,871 INFO [RS:2;asf911:52507] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager gracefully. 2018-06-21 23:23:42,881 INFO [RS:5;asf911:34033] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager gracefully. 2018-06-21 23:23:42,881 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1520): Closing 581b5ff62f5df0e6be8aaeb23aa08c64, disabling compactions & flushes 2018-06-21 23:23:42,882 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1560): Updates disabled for region hbase:namespace,,1529623395752.581b5ff62f5df0e6be8aaeb23aa08c64. 2018-06-21 23:23:42,882 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=78 B heapSize=232 B 2018-06-21 23:23:42,881 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1520): Closing b1a8d69ffdd43e80f30a919effdc7c4e, disabling compactions & flushes 2018-06-21 23:23:42,879 INFO [RS:5;asf911:50612] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2018-06-21 23:23:42,883 INFO [RS:5;asf911:50612] regionserver.HRegionServer(1069): stopping server asf911.gq1.ygridcore.net,50612,1529623419106 2018-06-21 23:23:42,883 DEBUG [RS:5;asf911:50612] zookeeper.MetaTableLocator(642): Stopping MetaTableLocator 2018-06-21 23:23:42,881 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(383): MemStoreFlusher.1 exiting 2018-06-21 23:23:42,879 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(383): MemStoreFlusher.0 exiting 2018-06-21 23:23:42,879 INFO [SplitLogWorker-asf911:34033] regionserver.SplitLogWorker(157): SplitLogWorker asf911.gq1.ygridcore.net,34033,1529623407773 exiting 2018-06-21 23:23:42,878 INFO [RS:3;asf911:56613] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager gracefully. 2018-06-21 23:23:42,883 INFO [RS:3;asf911:56613] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2018-06-21 23:23:42,878 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(383): MemStoreFlusher.1 exiting 2018-06-21 23:23:42,876 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(383): MemStoreFlusher.1 exiting 2018-06-21 23:23:42,874 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(383): MemStoreFlusher.0 exiting 2018-06-21 23:23:42,883 INFO [RS:5;asf911:50612] zookeeper.ReadOnlyZKClient(350): Close zookeeper connection 0x70e8353a to localhost:58989 2018-06-21 23:23:42,882 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1560): Updates disabled for region multi_test_table,nnn,1529623399386.b1a8d69ffdd43e80f30a919effdc7c4e. 2018-06-21 23:23:42,882 INFO [RS:5;asf911:34033] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2018-06-21 23:23:42,884 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1520): Closing 28b5f974b61112896f4c4f1127dc2b99, disabling compactions & flushes 2018-06-21 23:23:42,884 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1560): Updates disabled for region multi_test_table,mmm,1529623399386.28b5f974b61112896f4c4f1127dc2b99. 2018-06-21 23:23:42,882 INFO [RS:4;asf911:58235] regionserver.HRegionServer(1069): stopping server asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:42,885 DEBUG [RS:4;asf911:58235] zookeeper.MetaTableLocator(642): Stopping MetaTableLocator 2018-06-21 23:23:42,885 INFO [RS:4;asf911:58235] zookeeper.ReadOnlyZKClient(350): Close zookeeper connection 0x5690b764 to localhost:58989 2018-06-21 23:23:42,886 DEBUG [RS:4;asf911:58235] ipc.AbstractRpcClient(483): Stopping rpc client 2018-06-21 23:23:42,886 INFO [RS:4;asf911:58235] regionserver.HRegionServer(1399): Waiting on 10 regions to close 2018-06-21 23:23:42,882 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1520): Closing 7006d69958189f4e3b610c07f795d5d5, disabling compactions & flushes 2018-06-21 23:23:42,882 INFO [RS:2;asf911:52507] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2018-06-21 23:23:42,886 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1560): Updates disabled for region multi_test_table,yyy,1529623399386.7006d69958189f4e3b610c07f795d5d5. 2018-06-21 23:23:42,885 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1520): Closing c793c2c3948aad84865d30ebf71a529e, disabling compactions & flushes 2018-06-21 23:23:42,886 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=5.73 KB heapSize=12.20 KB 2018-06-21 23:23:42,885 INFO [RS:5;asf911:34033] regionserver.HRegionServer(1069): stopping server asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:42,887 DEBUG [RS:5;asf911:34033] zookeeper.MetaTableLocator(642): Stopping MetaTableLocator 2018-06-21 23:23:42,885 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=4.63 KB heapSize=9.70 KB 2018-06-21 23:23:42,887 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1520): Closing d4ac3b44d02d68507d6317e201672011, disabling compactions & flushes 2018-06-21 23:23:42,885 INFO [RS:3;asf911:56613] regionserver.HRegionServer(1069): stopping server asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:42,887 DEBUG [RS:3;asf911:56613] zookeeper.MetaTableLocator(642): Stopping MetaTableLocator 2018-06-21 23:23:42,885 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1520): Closing a657e5e90b9697e4cf81bc4bbde44800, disabling compactions & flushes 2018-06-21 23:23:42,885 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1520): Closing da9d0eefced1af8dfad2324a5d9e1bf7, disabling compactions & flushes 2018-06-21 23:23:42,884 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=4.10 KB heapSize=8.81 KB 2018-06-21 23:23:42,884 DEBUG [RS:5;asf911:50612] ipc.AbstractRpcClient(483): Stopping rpc client 2018-06-21 23:23:42,890 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1560): Updates disabled for region multi_test_table,fff,1529623399386.da9d0eefced1af8dfad2324a5d9e1bf7. 2018-06-21 23:23:42,888 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1560): Updates disabled for region multi_test_table,www,1529623399386.a657e5e90b9697e4cf81bc4bbde44800. 2018-06-21 23:23:42,887 INFO [RS:3;asf911:56613] zookeeper.ReadOnlyZKClient(350): Close zookeeper connection 0x3934cc14 to localhost:58989 2018-06-21 23:23:42,891 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=6.10 KB heapSize=13.28 KB 2018-06-21 23:23:42,887 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1560): Updates disabled for region multi_test_table,sss,1529623399386.d4ac3b44d02d68507d6317e201672011. 2018-06-21 23:23:42,887 INFO [RS:2;asf911:52507] regionserver.HRegionServer(1069): stopping server asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:42,892 DEBUG [RS:2;asf911:52507] zookeeper.MetaTableLocator(642): Stopping MetaTableLocator 2018-06-21 23:23:42,887 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1520): Closing 40503e5e9a757a4ccf579bf29b8d78c6, disabling compactions & flushes 2018-06-21 23:23:42,894 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1560): Updates disabled for region multi_test_table,uuu,1529623399386.40503e5e9a757a4ccf579bf29b8d78c6. 2018-06-21 23:23:42,887 INFO [RS:5;asf911:34033] zookeeper.ReadOnlyZKClient(350): Close zookeeper connection 0x0160fc1e to localhost:58989 2018-06-21 23:23:42,895 DEBUG [RS:5;asf911:34033] ipc.AbstractRpcClient(483): Stopping rpc client 2018-06-21 23:23:42,895 INFO [RS:5;asf911:34033] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-06-21 23:23:42,896 INFO [RS:5;asf911:34033] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-06-21 23:23:42,896 INFO [RS:5;asf911:34033] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-06-21 23:23:42,887 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1520): Closing 04d71925444c7c823938ba3d9ad0504d, disabling compactions & flushes 2018-06-21 23:23:42,886 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1560): Updates disabled for region multi_test_table,eee,1529623399386.c793c2c3948aad84865d30ebf71a529e. 2018-06-21 23:23:42,896 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1560): Updates disabled for region multi_test_table,iii,1529623399386.04d71925444c7c823938ba3d9ad0504d. 2018-06-21 23:23:42,896 INFO [RS:5;asf911:34033] regionserver.HRegionServer(1399): Waiting on 2 regions to close 2018-06-21 23:23:42,894 INFO [RS:2;asf911:52507] zookeeper.ReadOnlyZKClient(350): Close zookeeper connection 0x6e5097e4 to localhost:58989 2018-06-21 23:23:42,891 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=3.56 KB heapSize=7.51 KB 2018-06-21 23:23:42,891 DEBUG [RS:3;asf911:56613] ipc.AbstractRpcClient(483): Stopping rpc client 2018-06-21 23:23:42,891 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=5.73 KB heapSize=12.20 KB 2018-06-21 23:23:42,890 INFO [RS:5;asf911:50612] regionserver.HRegionServer(1097): stopping server asf911.gq1.ygridcore.net,50612,1529623419106; all regions closed. 2018-06-21 23:23:42,899 INFO [RS:3;asf911:56613] regionserver.HRegionServer(1399): Waiting on 8 regions to close 2018-06-21 23:23:42,899 DEBUG [RS:3;asf911:56613] regionserver.HRegionServer(1403): Online Regions={28b5f974b61112896f4c4f1127dc2b99=multi_test_table,mmm,1529623399386.28b5f974b61112896f4c4f1127dc2b99., da9d0eefced1af8dfad2324a5d9e1bf7=multi_test_table,fff,1529623399386.da9d0eefced1af8dfad2324a5d9e1bf7., a657e5e90b9697e4cf81bc4bbde44800=multi_test_table,www,1529623399386.a657e5e90b9697e4cf81bc4bbde44800., 77a32d153e5d8148ca9d599c6a3cb695=multi_test_table,xxx,1529623399386.77a32d153e5d8148ca9d599c6a3cb695., 1449fc3d526b1c6ab2fd158edec23277=multi_test_table,ooo,1529623399386.1449fc3d526b1c6ab2fd158edec23277., 0dfb3ebb8c759d632ec793ac0387ec95=multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95., 36d0a953419e9f81636aebf73e7ed5df=multi_test_table,ppp,1529623399386.36d0a953419e9f81636aebf73e7ed5df., 10cdebbb720d98123807841a0febf405=multi_test_table,ggg,1529623399386.10cdebbb720d98123807841a0febf405.} 2018-06-21 23:23:42,898 DEBUG [RS:2;asf911:52507] ipc.AbstractRpcClient(483): Stopping rpc client 2018-06-21 23:23:42,899 INFO [RS:2;asf911:52507] regionserver.HRegionServer(1399): Waiting on 8 regions to close 2018-06-21 23:23:42,898 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=3.63 KB heapSize=7.72 KB 2018-06-21 23:23:42,898 DEBUG [RS_CLOSE_META-regionserver/asf911:0-0] regionserver.HRegion(1520): Closing 1588230740, disabling compactions & flushes 2018-06-21 23:23:42,896 DEBUG [RS:5;asf911:34033] regionserver.HRegionServer(1403): Online Regions={c793c2c3948aad84865d30ebf71a529e=multi_test_table,eee,1529623399386.c793c2c3948aad84865d30ebf71a529e., 1588230740=hbase:meta,,1.1588230740} 2018-06-21 23:23:42,896 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=4.10 KB heapSize=8.81 KB 2018-06-21 23:23:42,896 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=3.81 KB heapSize=8.26 KB 2018-06-21 23:23:42,900 DEBUG [RS_CLOSE_META-regionserver/asf911:0-0] regionserver.HRegion(1560): Updates disabled for region hbase:meta,,1.1588230740 2018-06-21 23:23:42,900 DEBUG [RS:2;asf911:52507] regionserver.HRegionServer(1403): Online Regions={04d71925444c7c823938ba3d9ad0504d=multi_test_table,iii,1529623399386.04d71925444c7c823938ba3d9ad0504d., 40503e5e9a757a4ccf579bf29b8d78c6=multi_test_table,uuu,1529623399386.40503e5e9a757a4ccf579bf29b8d78c6., d4ac3b44d02d68507d6317e201672011=multi_test_table,sss,1529623399386.d4ac3b44d02d68507d6317e201672011., 77885b472341ce945c76aa06a4627146=multi_test_table,lll,1529623399386.77885b472341ce945c76aa06a4627146., dff6ab063ea78552b5e3c3ae7760ca5a=multi_test_table,ttt,1529623399386.dff6ab063ea78552b5e3c3ae7760ca5a., e66f718b26d1c4420497ccfb42f08321=multi_test_table,ddd,1529623399386.e66f718b26d1c4420497ccfb42f08321., 124da0ae361b264a91ea989bcd12b459=multi_test_table,ccc,1529623399386.124da0ae361b264a91ea989bcd12b459., fee67ae5d4f2b2be35fa516877a8d173=multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173.} 2018-06-21 23:23:42,910 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741901_1077{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW]]} size 0 2018-06-21 23:23:42,913 INFO [regionserver/asf911:0.Chore.1] hbase.ScheduledChore(180): Chore: MemstoreFlusherChore was stopped 2018-06-21 23:23:42,914 INFO [regionserver/asf911:0.leaseChecker] regionserver.Leases(149): Closed leases 2018-06-21 23:23:42,922 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741901_1077 size 91 2018-06-21 23:23:42,925 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741901_1077 size 91 2018-06-21 23:23:42,925 DEBUG [RS_CLOSE_META-regionserver/asf911:0-0] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/meta/1588230740/recovered.edits/119.seqid, newMaxSeqId=119, maxSeqId=116 2018-06-21 23:23:42,928 DEBUG [RS_CLOSE_META-regionserver/asf911:0-0] coprocessor.CoprocessorHost(288): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-06-21 23:23:42,928 INFO [RS_CLOSE_META-regionserver/asf911:0-0] regionserver.HRegion(1674): Closed hbase:meta,,1.1588230740 2018-06-21 23:23:42,928 DEBUG [RS_CLOSE_META-regionserver/asf911:0-0] handler.CloseRegionHandler(124): Closed hbase:meta,,1.1588230740 2018-06-21 23:23:42,932 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741903_1079{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|FINALIZED]]} size 0 2018-06-21 23:23:42,932 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741903_1079{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|FINALIZED]]} size 0 2018-06-21 23:23:42,932 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741903_1079{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|FINALIZED]]} size 0 2018-06-21 23:23:42,935 DEBUG [RS:5;asf911:50612] wal.AbstractFSWAL(854): Moved 1 WAL file(s) to /user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/oldWALs 2018-06-21 23:23:42,935 INFO [RS:5;asf911:50612] wal.AbstractFSWAL(857): Closed WAL: AsyncFSWAL asf911.gq1.ygridcore.net%2C50612%2C1529623419106:(num 1529623420378) 2018-06-21 23:23:42,936 DEBUG [RS:5;asf911:50612] ipc.AbstractRpcClient(483): Stopping rpc client 2018-06-21 23:23:42,936 INFO [RS:5;asf911:50612] regionserver.Leases(149): Closed leases 2018-06-21 23:23:42,942 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/namespace/581b5ff62f5df0e6be8aaeb23aa08c64/.tmp/info/a1a7d55f56aa42b6960ba30dfaa961f1 2018-06-21 23:23:42,948 INFO [RS:5;asf911:50612] hbase.ChoreService(327): Chore service for: regionserver/asf911:0 had [[ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region asf911.gq1.ygridcore.net,50612,1529623419106 Period: 120000 Unit: MILLISECONDS]] on shutdown 2018-06-21 23:23:42,948 INFO [regionserver/asf911:0.logRoller] regionserver.LogRoller(212): LogRoller exiting. 2018-06-21 23:23:42,948 INFO [RS:5;asf911:50612] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-06-21 23:23:42,949 INFO [RS:5;asf911:50612] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-06-21 23:23:42,949 INFO [RS:5;asf911:50612] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-06-21 23:23:42,951 INFO [RS:5;asf911:50612] ipc.NettyRpcServer(143): Stopping server on /67.195.81.155:50612 2018-06-21 23:23:42,964 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741906_1082{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|FINALIZED]]} size 0 2018-06-21 23:23:42,964 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741906_1082{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|FINALIZED], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|FINALIZED]]} size 0 2018-06-21 23:23:42,964 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741905_1081{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|FINALIZED]]} size 0 2018-06-21 23:23:42,964 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741905_1081{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|FINALIZED]]} size 0 2018-06-21 23:23:42,965 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741906_1082 size 5991 2018-06-21 23:23:42,965 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741905_1081 size 5991 2018-06-21 23:23:42,966 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741909_1085{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW]]} size 0 2018-06-21 23:23:42,966 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:50612-0x16424a89dd40011, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,50612,1529623419106 2018-06-21 23:23:42,966 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,50612,1529623419106 2018-06-21 23:23:42,966 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:42,966 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:50612-0x16424a89dd40011, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:42,966 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,50612,1529623419106 2018-06-21 23:23:42,966 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,50612,1529623419106 2018-06-21 23:23:42,966 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,50612,1529623419106 2018-06-21 23:23:42,966 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,50612,1529623419106 2018-06-21 23:23:42,967 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:42,967 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:42,967 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=4.63 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/28b5f974b61112896f4c4f1127dc2b99/.tmp/family/0fc780c620b14e60ae1fd076abe7965e 2018-06-21 23:23:42,967 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:42,968 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=4.10 KB at sequenceid=22 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/b1a8d69ffdd43e80f30a919effdc7c4e/.tmp/family/eb2787dc1e5b4dea83b96de97d1b1ce8 2018-06-21 23:23:42,969 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741909_1085 size 5991 2018-06-21 23:23:42,969 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741909_1085 size 5991 2018-06-21 23:23:42,970 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741907_1083{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW]]} size 0 2018-06-21 23:23:42,970 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741904_1080{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW]]} size 0 2018-06-21 23:23:42,971 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/namespace/581b5ff62f5df0e6be8aaeb23aa08c64/.tmp/info/a1a7d55f56aa42b6960ba30dfaa961f1 as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/namespace/581b5ff62f5df0e6be8aaeb23aa08c64/info/a1a7d55f56aa42b6960ba30dfaa961f1 2018-06-21 23:23:42,971 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741904_1080{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW]]} size 0 2018-06-21 23:23:42,972 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741908_1084{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW]]} size 0 2018-06-21 23:23:42,972 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741908_1084{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW]]} size 0 2018-06-21 23:23:42,973 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=5.73 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/a657e5e90b9697e4cf81bc4bbde44800/.tmp/family/2ba60f6425e34d908da0d0fbdcac0b18 2018-06-21 23:23:42,973 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=5.73 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/7006d69958189f4e3b610c07f795d5d5/.tmp/family/999f41058aaf48ceb23fddac38d62685 2018-06-21 23:23:42,974 INFO [RS:5;asf911:50612] regionserver.HRegionServer(1153): Exiting; stopping=asf911.gq1.ygridcore.net,50612,1529623419106; zookeeper connection closed. 2018-06-21 23:23:42,974 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:42,975 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=3.56 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/d4ac3b44d02d68507d6317e201672011/.tmp/family/610a5fe7bffb436ca50d257e29d5ea6b 2018-06-21 23:23:42,975 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@7cb5671b] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(221): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@7cb5671b 2018-06-21 23:23:42,975 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=6.10 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/da9d0eefced1af8dfad2324a5d9e1bf7/.tmp/family/187fd1e4cb984c7597b3fdbe54dcd348 2018-06-21 23:23:42,975 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:42,977 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:42,977 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741904_1080 size 5991 2018-06-21 23:23:42,978 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741907_1083 size 6093 2018-06-21 23:23:42,978 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741907_1083 size 6093 2018-06-21 23:23:42,978 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741908_1084 size 5991 2018-06-21 23:23:42,979 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741912_1088{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW]]} size 5982 2018-06-21 23:23:42,979 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741910_1086{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW]]} size 5991 2018-06-21 23:23:42,980 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:42,980 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:42,980 INFO [Time-limited test-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(122): /hbase/rs/asf911.gq1.ygridcore.net,50612,1529623419106 znode expired, triggering replicatorRemoved event 2018-06-21 23:23:42,980 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:42,980 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741910_1086 size 5991 2018-06-21 23:23:42,981 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741910_1086 size 5991 2018-06-21 23:23:42,981 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741912_1088 size 5982 2018-06-21 23:23:42,981 INFO [RegionServerTracker-0] master.RegionServerTracker(159): RegionServer ephemeral node deleted, processing expiration [asf911.gq1.ygridcore.net,50612,1529623419106] 2018-06-21 23:23:42,981 INFO [RegionServerTracker-0] master.ServerManager(597): Cluster shutdown set; asf911.gq1.ygridcore.net,50612,1529623419106 expired; onlineServers=5 2018-06-21 23:23:42,981 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:42,982 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:42,983 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:42,983 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741911_1087{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW]]} size 6093 2018-06-21 23:23:42,983 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741911_1087 size 6093 2018-06-21 23:23:42,983 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for eb2787dc1e5b4dea83b96de97d1b1ce8 2018-06-21 23:23:42,983 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741912_1088 size 5982 2018-06-21 23:23:42,985 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/b1a8d69ffdd43e80f30a919effdc7c4e/.tmp/family/eb2787dc1e5b4dea83b96de97d1b1ce8 as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/b1a8d69ffdd43e80f30a919effdc7c4e/family/eb2787dc1e5b4dea83b96de97d1b1ce8 2018-06-21 23:23:42,986 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2ba60f6425e34d908da0d0fbdcac0b18 2018-06-21 23:23:42,987 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 999f41058aaf48ceb23fddac38d62685 2018-06-21 23:23:42,988 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 187fd1e4cb984c7597b3fdbe54dcd348 2018-06-21 23:23:42,988 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/7006d69958189f4e3b610c07f795d5d5/.tmp/family/999f41058aaf48ceb23fddac38d62685 as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/7006d69958189f4e3b610c07f795d5d5/family/999f41058aaf48ceb23fddac38d62685 2018-06-21 23:23:42,993 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/a657e5e90b9697e4cf81bc4bbde44800/.tmp/family/2ba60f6425e34d908da0d0fbdcac0b18 as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/a657e5e90b9697e4cf81bc4bbde44800/family/2ba60f6425e34d908da0d0fbdcac0b18 2018-06-21 23:23:42,994 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:42,994 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/da9d0eefced1af8dfad2324a5d9e1bf7/.tmp/family/187fd1e4cb984c7597b3fdbe54dcd348 as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/da9d0eefced1af8dfad2324a5d9e1bf7/family/187fd1e4cb984c7597b3fdbe54dcd348 2018-06-21 23:23:42,994 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741911_1087 size 6093 2018-06-21 23:23:42,994 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:42,995 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/namespace/581b5ff62f5df0e6be8aaeb23aa08c64/info/a1a7d55f56aa42b6960ba30dfaa961f1, entries=2, sequenceid=6, filesize=4.8 K 2018-06-21 23:23:42,995 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0fc780c620b14e60ae1fd076abe7965e 2018-06-21 23:23:42,996 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(2769): Finished flush of dataSize ~78 B/78, heapSize ~448 B/448, currentSize=0 B/0 for 581b5ff62f5df0e6be8aaeb23aa08c64 in 114ms, sequenceid=6, compaction requested=false 2018-06-21 23:23:43,000 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 610a5fe7bffb436ca50d257e29d5ea6b 2018-06-21 23:23:43,001 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/28b5f974b61112896f4c4f1127dc2b99/.tmp/family/0fc780c620b14e60ae1fd076abe7965e as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/28b5f974b61112896f4c4f1127dc2b99/family/0fc780c620b14e60ae1fd076abe7965e 2018-06-21 23:23:43,004 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/d4ac3b44d02d68507d6317e201672011/.tmp/family/610a5fe7bffb436ca50d257e29d5ea6b as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/d4ac3b44d02d68507d6317e201672011/family/610a5fe7bffb436ca50d257e29d5ea6b 2018-06-21 23:23:43,006 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for eb2787dc1e5b4dea83b96de97d1b1ce8 2018-06-21 23:23:43,006 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/b1a8d69ffdd43e80f30a919effdc7c4e/family/eb2787dc1e5b4dea83b96de97d1b1ce8, entries=30, sequenceid=22, filesize=5.9 K 2018-06-21 23:23:43,007 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/hbase/namespace/581b5ff62f5df0e6be8aaeb23aa08c64/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2018-06-21 23:23:43,008 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(2769): Finished flush of dataSize ~4.10 KB/4194, heapSize ~9.02 KB/9240, currentSize=0 B/0 for b1a8d69ffdd43e80f30a919effdc7c4e in 124ms, sequenceid=22, compaction requested=false 2018-06-21 23:23:43,008 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 999f41058aaf48ceb23fddac38d62685 2018-06-21 23:23:43,008 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/7006d69958189f4e3b610c07f795d5d5/family/999f41058aaf48ceb23fddac38d62685, entries=30, sequenceid=16, filesize=5.9 K 2018-06-21 23:23:43,016 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2ba60f6425e34d908da0d0fbdcac0b18 2018-06-21 23:23:43,018 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1674): Closed hbase:namespace,,1529623395752.581b5ff62f5df0e6be8aaeb23aa08c64. 2018-06-21 23:23:43,018 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] handler.CloseRegionHandler(124): Closed hbase:namespace,,1529623395752.581b5ff62f5df0e6be8aaeb23aa08c64. 2018-06-21 23:23:43,019 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1520): Closing 91ab16f39c84b448bdc2e5a1e1fc801b, disabling compactions & flushes 2018-06-21 23:23:43,019 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1560): Updates disabled for region multi_test_table,hhh,1529623399386.91ab16f39c84b448bdc2e5a1e1fc801b. 2018-06-21 23:23:43,019 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/a657e5e90b9697e4cf81bc4bbde44800/family/2ba60f6425e34d908da0d0fbdcac0b18, entries=30, sequenceid=16, filesize=5.9 K 2018-06-21 23:23:43,019 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=3.81 KB heapSize=8.26 KB 2018-06-21 23:23:43,024 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(2769): Finished flush of dataSize ~5.73 KB/5871, heapSize ~12.41 KB/12712, currentSize=0 B/0 for 7006d69958189f4e3b610c07f795d5d5 in 138ms, sequenceid=16, compaction requested=false 2018-06-21 23:23:43,026 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 187fd1e4cb984c7597b3fdbe54dcd348 2018-06-21 23:23:43,026 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(2769): Finished flush of dataSize ~5.73 KB/5871, heapSize ~12.41 KB/12712, currentSize=0 B/0 for a657e5e90b9697e4cf81bc4bbde44800 in 135ms, sequenceid=16, compaction requested=false 2018-06-21 23:23:43,026 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/da9d0eefced1af8dfad2324a5d9e1bf7/family/187fd1e4cb984c7597b3fdbe54dcd348, entries=33, sequenceid=16, filesize=6.0 K 2018-06-21 23:23:43,027 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(2769): Finished flush of dataSize ~6.10 KB/6243, heapSize ~13.49 KB/13816, currentSize=0 B/0 for da9d0eefced1af8dfad2324a5d9e1bf7 in 136ms, sequenceid=16, compaction requested=false 2018-06-21 23:23:43,032 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 610a5fe7bffb436ca50d257e29d5ea6b 2018-06-21 23:23:43,032 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/d4ac3b44d02d68507d6317e201672011/family/610a5fe7bffb436ca50d257e29d5ea6b, entries=30, sequenceid=18, filesize=5.9 K 2018-06-21 23:23:43,035 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0fc780c620b14e60ae1fd076abe7965e 2018-06-21 23:23:43,036 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(2769): Finished flush of dataSize ~3.56 KB/3648, heapSize ~7.93 KB/8120, currentSize=0 B/0 for d4ac3b44d02d68507d6317e201672011 in 144ms, sequenceid=18, compaction requested=false 2018-06-21 23:23:43,036 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/b1a8d69ffdd43e80f30a919effdc7c4e/recovered.edits/25.seqid, newMaxSeqId=25, maxSeqId=10 2018-06-21 23:23:43,036 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/28b5f974b61112896f4c4f1127dc2b99/family/0fc780c620b14e60ae1fd076abe7965e, entries=30, sequenceid=18, filesize=5.9 K 2018-06-21 23:23:43,037 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/da9d0eefced1af8dfad2324a5d9e1bf7/recovered.edits/19.seqid, newMaxSeqId=19, maxSeqId=1 2018-06-21 23:23:43,037 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(2769): Finished flush of dataSize ~4.63 KB/4740, heapSize ~10.12 KB/10360, currentSize=0 B/0 for 28b5f974b61112896f4c4f1127dc2b99 in 152ms, sequenceid=18, compaction requested=false 2018-06-21 23:23:43,045 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1674): Closed multi_test_table,nnn,1529623399386.b1a8d69ffdd43e80f30a919effdc7c4e. 2018-06-21 23:23:43,048 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] handler.CloseRegionHandler(124): Closed multi_test_table,nnn,1529623399386.b1a8d69ffdd43e80f30a919effdc7c4e. 2018-06-21 23:23:43,048 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1520): Closing e04a2e293b4c1bbd21e87d569c11154e, disabling compactions & flushes 2018-06-21 23:23:43,048 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1560): Updates disabled for region multi_test_table,bbb,1529623399386.e04a2e293b4c1bbd21e87d569c11154e. 2018-06-21 23:23:43,048 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=6.10 KB heapSize=13.28 KB 2018-06-21 23:23:43,050 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1674): Closed multi_test_table,fff,1529623399386.da9d0eefced1af8dfad2324a5d9e1bf7. 2018-06-21 23:23:43,051 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] handler.CloseRegionHandler(124): Closed multi_test_table,fff,1529623399386.da9d0eefced1af8dfad2324a5d9e1bf7. 2018-06-21 23:23:43,051 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1520): Closing 77a32d153e5d8148ca9d599c6a3cb695, disabling compactions & flushes 2018-06-21 23:23:43,051 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1560): Updates disabled for region multi_test_table,xxx,1529623399386.77a32d153e5d8148ca9d599c6a3cb695. 2018-06-21 23:23:43,051 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/a657e5e90b9697e4cf81bc4bbde44800/recovered.edits/19.seqid, newMaxSeqId=19, maxSeqId=1 2018-06-21 23:23:43,051 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=8.31 KB heapSize=20.91 KB 2018-06-21 23:23:43,055 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/7006d69958189f4e3b610c07f795d5d5/recovered.edits/19.seqid, newMaxSeqId=19, maxSeqId=1 2018-06-21 23:23:43,059 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1674): Closed multi_test_table,yyy,1529623399386.7006d69958189f4e3b610c07f795d5d5. 2018-06-21 23:23:43,060 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] handler.CloseRegionHandler(124): Closed multi_test_table,yyy,1529623399386.7006d69958189f4e3b610c07f795d5d5. 2018-06-21 23:23:43,060 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1520): Closing 8e2537011449fca6295a357cbde76e6c, disabling compactions & flushes 2018-06-21 23:23:43,060 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1560): Updates disabled for region multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c. 2018-06-21 23:23:43,060 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=4.10 KB heapSize=8.81 KB 2018-06-21 23:23:43,066 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1674): Closed multi_test_table,www,1529623399386.a657e5e90b9697e4cf81bc4bbde44800. 2018-06-21 23:23:43,067 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] handler.CloseRegionHandler(124): Closed multi_test_table,www,1529623399386.a657e5e90b9697e4cf81bc4bbde44800. 2018-06-21 23:23:43,067 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1520): Closing 1449fc3d526b1c6ab2fd158edec23277, disabling compactions & flushes 2018-06-21 23:23:43,067 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1560): Updates disabled for region multi_test_table,ooo,1529623399386.1449fc3d526b1c6ab2fd158edec23277. 2018-06-21 23:23:43,067 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=5.70 KB heapSize=12.09 KB 2018-06-21 23:23:43,084 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/d4ac3b44d02d68507d6317e201672011/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=7 2018-06-21 23:23:43,085 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/28b5f974b61112896f4c4f1127dc2b99/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=5 2018-06-21 23:23:43,088 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741913_1089{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|FINALIZED]]} size 0 2018-06-21 23:23:43,088 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741913_1089{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|FINALIZED], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|FINALIZED]]} size 0 2018-06-21 23:23:43,096 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1674): Closed multi_test_table,sss,1529623399386.d4ac3b44d02d68507d6317e201672011. 2018-06-21 23:23:43,096 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1674): Closed multi_test_table,mmm,1529623399386.28b5f974b61112896f4c4f1127dc2b99. 2018-06-21 23:23:43,096 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] handler.CloseRegionHandler(124): Closed multi_test_table,sss,1529623399386.d4ac3b44d02d68507d6317e201672011. 2018-06-21 23:23:43,096 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] handler.CloseRegionHandler(124): Closed multi_test_table,mmm,1529623399386.28b5f974b61112896f4c4f1127dc2b99. 2018-06-21 23:23:43,097 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1520): Closing 0dfb3ebb8c759d632ec793ac0387ec95, disabling compactions & flushes 2018-06-21 23:23:43,097 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1520): Closing 77885b472341ce945c76aa06a4627146, disabling compactions & flushes 2018-06-21 23:23:43,097 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1560): Updates disabled for region multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95. 2018-06-21 23:23:43,097 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1560): Updates disabled for region multi_test_table,lll,1529623399386.77885b472341ce945c76aa06a4627146. 2018-06-21 23:23:43,097 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=3.60 KB heapSize=7.62 KB 2018-06-21 23:23:43,097 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=5.70 KB heapSize=12.09 KB 2018-06-21 23:23:43,101 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=3.81 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/91ab16f39c84b448bdc2e5a1e1fc801b/.tmp/family/1e02b0e973684383acbc889edfc4bee9 2018-06-21 23:23:43,101 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741913_1089 size 6093 2018-06-21 23:23:43,116 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 1e02b0e973684383acbc889edfc4bee9 2018-06-21 23:23:43,118 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/91ab16f39c84b448bdc2e5a1e1fc801b/.tmp/family/1e02b0e973684383acbc889edfc4bee9 as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/91ab16f39c84b448bdc2e5a1e1fc801b/family/1e02b0e973684383acbc889edfc4bee9 2018-06-21 23:23:43,132 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 1e02b0e973684383acbc889edfc4bee9 2018-06-21 23:23:43,132 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/91ab16f39c84b448bdc2e5a1e1fc801b/family/1e02b0e973684383acbc889edfc4bee9, entries=33, sequenceid=18, filesize=6.0 K 2018-06-21 23:23:43,134 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(2769): Finished flush of dataSize ~3.81 KB/3903, heapSize ~8.68 KB/8888, currentSize=0 B/0 for 91ab16f39c84b448bdc2e5a1e1fc801b in 115ms, sequenceid=18, compaction requested=false 2018-06-21 23:23:43,161 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741914_1090{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|FINALIZED]]} size 0 2018-06-21 23:23:43,161 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741914_1090{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|FINALIZED], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|FINALIZED]]} size 0 2018-06-21 23:23:43,161 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741914_1090{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|FINALIZED], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|FINALIZED], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|FINALIZED]]} size 0 2018-06-21 23:23:43,180 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=6.10 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/e04a2e293b4c1bbd21e87d569c11154e/.tmp/family/69b545c998044d9f9af0076ee908fb9a 2018-06-21 23:23:43,182 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741915_1091{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:43,182 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741915_1091{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:43,183 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741915_1091 size 5991 2018-06-21 23:23:43,184 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=5.70 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/1449fc3d526b1c6ab2fd158edec23277/.tmp/family/004944921d48412cb739d132c9332672 2018-06-21 23:23:43,186 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741916_1092{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:43,187 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741916_1092{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:43,187 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741916_1092{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:43,189 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 69b545c998044d9f9af0076ee908fb9a 2018-06-21 23:23:43,190 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/e04a2e293b4c1bbd21e87d569c11154e/.tmp/family/69b545c998044d9f9af0076ee908fb9a as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/e04a2e293b4c1bbd21e87d569c11154e/family/69b545c998044d9f9af0076ee908fb9a 2018-06-21 23:23:43,192 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/91ab16f39c84b448bdc2e5a1e1fc801b/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=7 2018-06-21 23:23:43,193 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=8.31 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/77a32d153e5d8148ca9d599c6a3cb695/.tmp/family/18f9547250634b6990c5876ef054bf4d 2018-06-21 23:23:43,194 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1674): Closed multi_test_table,hhh,1529623399386.91ab16f39c84b448bdc2e5a1e1fc801b. 2018-06-21 23:23:43,194 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] handler.CloseRegionHandler(124): Closed multi_test_table,hhh,1529623399386.91ab16f39c84b448bdc2e5a1e1fc801b. 2018-06-21 23:23:43,194 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1520): Closing cb0df54f72fae3cd39a03514dbbaa19c, disabling compactions & flushes 2018-06-21 23:23:43,194 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1560): Updates disabled for region multi_test_table,,1529623399386.cb0df54f72fae3cd39a03514dbbaa19c. 2018-06-21 23:23:43,194 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=3.10 KB heapSize=5.20 KB 2018-06-21 23:23:43,194 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 004944921d48412cb739d132c9332672 2018-06-21 23:23:43,196 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/1449fc3d526b1c6ab2fd158edec23277/.tmp/family/004944921d48412cb739d132c9332672 as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/1449fc3d526b1c6ab2fd158edec23277/family/004944921d48412cb739d132c9332672 2018-06-21 23:23:43,206 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 18f9547250634b6990c5876ef054bf4d 2018-06-21 23:23:43,207 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741918_1094{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW]]} size 0 2018-06-21 23:23:43,208 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/77a32d153e5d8148ca9d599c6a3cb695/.tmp/family/18f9547250634b6990c5876ef054bf4d as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/77a32d153e5d8148ca9d599c6a3cb695/family/18f9547250634b6990c5876ef054bf4d 2018-06-21 23:23:43,208 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=4.10 KB at sequenceid=24 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/.tmp/family/fbd43cda7e28499d95767b756e30d18f 2018-06-21 23:23:43,209 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741918_1094 size 5991 2018-06-21 23:23:43,210 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741918_1094 size 5991 2018-06-21 23:23:43,212 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741917_1093{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:43,212 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741919_1095{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW]]} size 0 2018-06-21 23:23:43,213 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741919_1095{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|FINALIZED]]} size 0 2018-06-21 23:23:43,213 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741919_1095{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|FINALIZED]]} size 0 2018-06-21 23:23:43,214 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741917_1093{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:43,215 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 69b545c998044d9f9af0076ee908fb9a 2018-06-21 23:23:43,215 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/e04a2e293b4c1bbd21e87d569c11154e/family/69b545c998044d9f9af0076ee908fb9a, entries=33, sequenceid=16, filesize=6.0 K 2018-06-21 23:23:43,216 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=3.60 KB at sequenceid=20 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/.tmp/family/3730cf52c860405eba2e2ffc5c7b7973 2018-06-21 23:23:43,216 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741917_1093{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:43,217 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(2769): Finished flush of dataSize ~6.10 KB/6243, heapSize ~13.49 KB/13816, currentSize=0 B/0 for e04a2e293b4c1bbd21e87d569c11154e in 169ms, sequenceid=16, compaction requested=false 2018-06-21 23:23:43,218 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=5.70 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/77885b472341ce945c76aa06a4627146/.tmp/family/d3b8f36378a34ab0b1b14c7bcd3f4b80 2018-06-21 23:23:43,223 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for fbd43cda7e28499d95767b756e30d18f 2018-06-21 23:23:43,225 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 004944921d48412cb739d132c9332672 2018-06-21 23:23:43,225 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/1449fc3d526b1c6ab2fd158edec23277/family/004944921d48412cb739d132c9332672, entries=30, sequenceid=16, filesize=5.9 K 2018-06-21 23:23:43,226 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 18f9547250634b6990c5876ef054bf4d 2018-06-21 23:23:43,226 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/77a32d153e5d8148ca9d599c6a3cb695/family/18f9547250634b6990c5876ef054bf4d, entries=131, sequenceid=48, filesize=10.1 K 2018-06-21 23:23:43,226 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(2769): Finished flush of dataSize ~5.70 KB/5832, heapSize ~12.30 KB/12600, currentSize=0 B/0 for 1449fc3d526b1c6ab2fd158edec23277 in 159ms, sequenceid=16, compaction requested=false 2018-06-21 23:23:43,227 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/.tmp/family/fbd43cda7e28499d95767b756e30d18f as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/family/fbd43cda7e28499d95767b756e30d18f 2018-06-21 23:23:43,228 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(2769): Finished flush of dataSize ~8.31 KB/8505, heapSize ~21.12 KB/21624, currentSize=0 B/0 for 77a32d153e5d8148ca9d599c6a3cb695 in 177ms, sequenceid=48, compaction requested=false 2018-06-21 23:23:43,230 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d3b8f36378a34ab0b1b14c7bcd3f4b80 2018-06-21 23:23:43,230 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3730cf52c860405eba2e2ffc5c7b7973 2018-06-21 23:23:43,234 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/e04a2e293b4c1bbd21e87d569c11154e/recovered.edits/19.seqid, newMaxSeqId=19, maxSeqId=1 2018-06-21 23:23:43,235 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/.tmp/family/3730cf52c860405eba2e2ffc5c7b7973 as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/family/3730cf52c860405eba2e2ffc5c7b7973 2018-06-21 23:23:43,236 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1674): Closed multi_test_table,bbb,1529623399386.e04a2e293b4c1bbd21e87d569c11154e. 2018-06-21 23:23:43,235 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/77885b472341ce945c76aa06a4627146/.tmp/family/d3b8f36378a34ab0b1b14c7bcd3f4b80 as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/77885b472341ce945c76aa06a4627146/family/d3b8f36378a34ab0b1b14c7bcd3f4b80 2018-06-21 23:23:43,236 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] handler.CloseRegionHandler(124): Closed multi_test_table,bbb,1529623399386.e04a2e293b4c1bbd21e87d569c11154e. 2018-06-21 23:23:43,236 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1520): Closing 573e96916daa12dfc083e39a253dc3a2, disabling compactions & flushes 2018-06-21 23:23:43,236 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1560): Updates disabled for region multi_test_table,jjj,1529623399386.573e96916daa12dfc083e39a253dc3a2. 2018-06-21 23:23:43,236 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=4.63 KB heapSize=9.70 KB 2018-06-21 23:23:43,238 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741920_1096{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW]]} size 0 2018-06-21 23:23:43,238 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741920_1096{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|FINALIZED]]} size 0 2018-06-21 23:23:43,239 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741920_1096{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|FINALIZED]]} size 0 2018-06-21 23:23:43,240 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=3.10 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/cb0df54f72fae3cd39a03514dbbaa19c/.tmp/family/c54475e684024d5f81f5a4b53b6c2b35 2018-06-21 23:23:43,244 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/1449fc3d526b1c6ab2fd158edec23277/recovered.edits/19.seqid, newMaxSeqId=19, maxSeqId=1 2018-06-21 23:23:43,246 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1674): Closed multi_test_table,ooo,1529623399386.1449fc3d526b1c6ab2fd158edec23277. 2018-06-21 23:23:43,246 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for fbd43cda7e28499d95767b756e30d18f 2018-06-21 23:23:43,247 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] handler.CloseRegionHandler(124): Closed multi_test_table,ooo,1529623399386.1449fc3d526b1c6ab2fd158edec23277. 2018-06-21 23:23:43,247 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1520): Closing 36d0a953419e9f81636aebf73e7ed5df, disabling compactions & flushes 2018-06-21 23:23:43,247 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1560): Updates disabled for region multi_test_table,ppp,1529623399386.36d0a953419e9f81636aebf73e7ed5df. 2018-06-21 23:23:43,247 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/77a32d153e5d8148ca9d599c6a3cb695/recovered.edits/51.seqid, newMaxSeqId=51, maxSeqId=14 2018-06-21 23:23:43,247 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=3.61 KB heapSize=7.63 KB 2018-06-21 23:23:43,247 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/family/fbd43cda7e28499d95767b756e30d18f, entries=30, sequenceid=24, filesize=5.9 K 2018-06-21 23:23:43,248 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1674): Closed multi_test_table,xxx,1529623399386.77a32d153e5d8148ca9d599c6a3cb695. 2018-06-21 23:23:43,248 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] handler.CloseRegionHandler(124): Closed multi_test_table,xxx,1529623399386.77a32d153e5d8148ca9d599c6a3cb695. 2018-06-21 23:23:43,248 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1520): Closing 10cdebbb720d98123807841a0febf405, disabling compactions & flushes 2018-06-21 23:23:43,249 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1560): Updates disabled for region multi_test_table,ggg,1529623399386.10cdebbb720d98123807841a0febf405. 2018-06-21 23:23:43,249 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=6.10 KB heapSize=13.28 KB 2018-06-21 23:23:43,249 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(2769): Finished flush of dataSize ~4.10 KB/4194, heapSize ~9.02 KB/9240, currentSize=0 B/0 for 8e2537011449fca6295a357cbde76e6c in 189ms, sequenceid=24, compaction requested=true 2018-06-21 23:23:43,249 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c54475e684024d5f81f5a4b53b6c2b35 2018-06-21 23:23:43,254 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 3730cf52c860405eba2e2ffc5c7b7973 2018-06-21 23:23:43,254 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/cb0df54f72fae3cd39a03514dbbaa19c/.tmp/family/c54475e684024d5f81f5a4b53b6c2b35 as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/cb0df54f72fae3cd39a03514dbbaa19c/family/c54475e684024d5f81f5a4b53b6c2b35 2018-06-21 23:23:43,255 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/family/3730cf52c860405eba2e2ffc5c7b7973, entries=30, sequenceid=20, filesize=5.9 K 2018-06-21 23:23:43,258 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(2769): Finished flush of dataSize ~3.60 KB/3687, heapSize ~8.04 KB/8232, currentSize=0 B/0 for 0dfb3ebb8c759d632ec793ac0387ec95 in 160ms, sequenceid=20, compaction requested=true 2018-06-21 23:23:43,261 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d3b8f36378a34ab0b1b14c7bcd3f4b80 2018-06-21 23:23:43,261 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/77885b472341ce945c76aa06a4627146/family/d3b8f36378a34ab0b1b14c7bcd3f4b80, entries=30, sequenceid=16, filesize=5.9 K 2018-06-21 23:23:43,262 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(2769): Finished flush of dataSize ~5.70 KB/5832, heapSize ~12.30 KB/12600, currentSize=0 B/0 for 77885b472341ce945c76aa06a4627146 in 165ms, sequenceid=16, compaction requested=false 2018-06-21 23:23:43,267 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/8e2537011449fca6295a357cbde76e6c/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=12 2018-06-21 23:23:43,268 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/0dfb3ebb8c759d632ec793ac0387ec95/recovered.edits/23.seqid, newMaxSeqId=23, maxSeqId=9 2018-06-21 23:23:43,269 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741921_1097{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW]]} size 0 2018-06-21 23:23:43,269 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741921_1097{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|FINALIZED]]} size 0 2018-06-21 23:23:43,270 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1674): Closed multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95. 2018-06-21 23:23:43,270 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] handler.CloseRegionHandler(124): Closed multi_test_table,vvv,1529623399386.0dfb3ebb8c759d632ec793ac0387ec95. 2018-06-21 23:23:43,273 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1674): Closed multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c. 2018-06-21 23:23:43,274 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] handler.CloseRegionHandler(124): Closed multi_test_table,rrr,1529623399386.8e2537011449fca6295a357cbde76e6c. 2018-06-21 23:23:43,274 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=4.63 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/573e96916daa12dfc083e39a253dc3a2/.tmp/family/39cf187fee35451cb244ae0bfb497d4a 2018-06-21 23:23:43,274 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1520): Closing 7480a111a1f600aa34b97a77e47640bc, disabling compactions & flushes 2018-06-21 23:23:43,274 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741921_1097 size 5991 2018-06-21 23:23:43,274 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1560): Updates disabled for region multi_test_table,qqq,1529623399386.7480a111a1f600aa34b97a77e47640bc. 2018-06-21 23:23:43,274 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=5.73 KB heapSize=12.20 KB 2018-06-21 23:23:43,280 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c54475e684024d5f81f5a4b53b6c2b35 2018-06-21 23:23:43,280 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/cb0df54f72fae3cd39a03514dbbaa19c/family/c54475e684024d5f81f5a4b53b6c2b35, entries=14, sequenceid=17, filesize=5.3 K 2018-06-21 23:23:43,282 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(2769): Finished flush of dataSize ~3.10 KB/3178, heapSize ~5.41 KB/5544, currentSize=0 B/0 for cb0df54f72fae3cd39a03514dbbaa19c in 88ms, sequenceid=17, compaction requested=false 2018-06-21 23:23:43,283 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/77885b472341ce945c76aa06a4627146/recovered.edits/19.seqid, newMaxSeqId=19, maxSeqId=1 2018-06-21 23:23:43,285 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1674): Closed multi_test_table,lll,1529623399386.77885b472341ce945c76aa06a4627146. 2018-06-21 23:23:43,285 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] handler.CloseRegionHandler(124): Closed multi_test_table,lll,1529623399386.77885b472341ce945c76aa06a4627146. 2018-06-21 23:23:43,285 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1520): Closing dff6ab063ea78552b5e3c3ae7760ca5a, disabling compactions & flushes 2018-06-21 23:23:43,285 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1560): Updates disabled for region multi_test_table,ttt,1529623399386.dff6ab063ea78552b5e3c3ae7760ca5a. 2018-06-21 23:23:43,286 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=5.70 KB heapSize=12.09 KB 2018-06-21 23:23:43,302 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741923_1099{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|FINALIZED]]} size 0 2018-06-21 23:23:43,303 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741923_1099{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|FINALIZED], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|FINALIZED]]} size 0 2018-06-21 23:23:43,307 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741923_1099{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|FINALIZED], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|FINALIZED], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|FINALIZED]]} size 0 2018-06-21 23:23:43,309 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 39cf187fee35451cb244ae0bfb497d4a 2018-06-21 23:23:43,309 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=3.61 KB at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/36d0a953419e9f81636aebf73e7ed5df/.tmp/family/ce17e60d2cfe4df3832f450492e179ea 2018-06-21 23:23:43,310 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741922_1098{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|FINALIZED]]} size 0 2018-06-21 23:23:43,310 INFO [regionserver/asf911:0.leaseChecker] regionserver.Leases(149): Closed leases 2018-06-21 23:23:43,310 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741922_1098{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|FINALIZED]]} size 0 2018-06-21 23:23:43,311 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741922_1098{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|FINALIZED]]} size 0 2018-06-21 23:23:43,313 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=6.10 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/10cdebbb720d98123807841a0febf405/.tmp/family/2478f7e4b40040f9a97894d1d8cfece5 2018-06-21 23:23:43,315 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/573e96916daa12dfc083e39a253dc3a2/.tmp/family/39cf187fee35451cb244ae0bfb497d4a as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/573e96916daa12dfc083e39a253dc3a2/family/39cf187fee35451cb244ae0bfb497d4a 2018-06-21 23:23:43,318 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/cb0df54f72fae3cd39a03514dbbaa19c/recovered.edits/20.seqid, newMaxSeqId=20, maxSeqId=1 2018-06-21 23:23:43,319 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1674): Closed multi_test_table,,1529623399386.cb0df54f72fae3cd39a03514dbbaa19c. 2018-06-21 23:23:43,319 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] handler.CloseRegionHandler(124): Closed multi_test_table,,1529623399386.cb0df54f72fae3cd39a03514dbbaa19c. 2018-06-21 23:23:43,320 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1520): Closing 794028a2b8fc0a516890763306629d4c, disabling compactions & flushes 2018-06-21 23:23:43,320 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1560): Updates disabled for region multi_test_table,zzz,1529623399386.794028a2b8fc0a516890763306629d4c. 2018-06-21 23:23:43,320 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=1.79 KB heapSize=5.60 KB 2018-06-21 23:23:43,323 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2478f7e4b40040f9a97894d1d8cfece5 2018-06-21 23:23:43,325 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for ce17e60d2cfe4df3832f450492e179ea 2018-06-21 23:23:43,326 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/10cdebbb720d98123807841a0febf405/.tmp/family/2478f7e4b40040f9a97894d1d8cfece5 as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/10cdebbb720d98123807841a0febf405/family/2478f7e4b40040f9a97894d1d8cfece5 2018-06-21 23:23:43,328 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/36d0a953419e9f81636aebf73e7ed5df/.tmp/family/ce17e60d2cfe4df3832f450492e179ea as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/36d0a953419e9f81636aebf73e7ed5df/family/ce17e60d2cfe4df3832f450492e179ea 2018-06-21 23:23:43,328 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 39cf187fee35451cb244ae0bfb497d4a 2018-06-21 23:23:43,328 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/573e96916daa12dfc083e39a253dc3a2/family/39cf187fee35451cb244ae0bfb497d4a, entries=30, sequenceid=18, filesize=5.9 K 2018-06-21 23:23:43,329 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741924_1100{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:43,329 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741924_1100{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|FINALIZED]]} size 0 2018-06-21 23:23:43,331 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(2769): Finished flush of dataSize ~4.63 KB/4740, heapSize ~10.12 KB/10360, currentSize=0 B/0 for 573e96916daa12dfc083e39a253dc3a2 in 95ms, sequenceid=18, compaction requested=false 2018-06-21 23:23:43,332 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741924_1100 size 6031 2018-06-21 23:23:43,338 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2478f7e4b40040f9a97894d1d8cfece5 2018-06-21 23:23:43,339 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/10cdebbb720d98123807841a0febf405/family/2478f7e4b40040f9a97894d1d8cfece5, entries=33, sequenceid=16, filesize=6.0 K 2018-06-21 23:23:43,339 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for ce17e60d2cfe4df3832f450492e179ea 2018-06-21 23:23:43,339 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/36d0a953419e9f81636aebf73e7ed5df/family/ce17e60d2cfe4df3832f450492e179ea, entries=31, sequenceid=19, filesize=5.9 K 2018-06-21 23:23:43,345 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=5.73 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/7480a111a1f600aa34b97a77e47640bc/.tmp/family/2f9eee0859de402eb910b10632428c02 2018-06-21 23:23:43,360 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(2769): Finished flush of dataSize ~6.10 KB/6243, heapSize ~13.49 KB/13816, currentSize=0 B/0 for 10cdebbb720d98123807841a0febf405 in 110ms, sequenceid=16, compaction requested=false 2018-06-21 23:23:43,360 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(2769): Finished flush of dataSize ~3.61 KB/3692, heapSize ~8.05 KB/8240, currentSize=0 B/0 for 36d0a953419e9f81636aebf73e7ed5df in 113ms, sequenceid=19, compaction requested=false 2018-06-21 23:23:43,361 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741925_1101{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:43,361 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741925_1101{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|FINALIZED]]} size 0 2018-06-21 23:23:43,366 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741925_1101 size 5991 2018-06-21 23:23:43,370 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=5.70 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/dff6ab063ea78552b5e3c3ae7760ca5a/.tmp/family/45cf8cdb5c344239afe0b620f5c1ebb6 2018-06-21 23:23:43,370 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2f9eee0859de402eb910b10632428c02 2018-06-21 23:23:43,372 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/7480a111a1f600aa34b97a77e47640bc/.tmp/family/2f9eee0859de402eb910b10632428c02 as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/7480a111a1f600aa34b97a77e47640bc/family/2f9eee0859de402eb910b10632428c02 2018-06-21 23:23:43,373 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/573e96916daa12dfc083e39a253dc3a2/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=5 2018-06-21 23:23:43,375 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1674): Closed multi_test_table,jjj,1529623399386.573e96916daa12dfc083e39a253dc3a2. 2018-06-21 23:23:43,375 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741926_1102{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|FINALIZED]]} size 0 2018-06-21 23:23:43,375 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] handler.CloseRegionHandler(124): Closed multi_test_table,jjj,1529623399386.573e96916daa12dfc083e39a253dc3a2. 2018-06-21 23:23:43,376 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741926_1102{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|FINALIZED]]} size 0 2018-06-21 23:23:43,376 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741926_1102{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|FINALIZED]]} size 0 2018-06-21 23:23:43,377 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/36d0a953419e9f81636aebf73e7ed5df/recovered.edits/22.seqid, newMaxSeqId=22, maxSeqId=7 2018-06-21 23:23:43,377 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=1.79 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/794028a2b8fc0a516890763306629d4c/.tmp/family/bc342c2523d744a58c14d1175d2fc967 2018-06-21 23:23:43,377 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/10cdebbb720d98123807841a0febf405/recovered.edits/19.seqid, newMaxSeqId=19, maxSeqId=1 2018-06-21 23:23:43,378 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1674): Closed multi_test_table,ppp,1529623399386.36d0a953419e9f81636aebf73e7ed5df. 2018-06-21 23:23:43,378 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] handler.CloseRegionHandler(124): Closed multi_test_table,ppp,1529623399386.36d0a953419e9f81636aebf73e7ed5df. 2018-06-21 23:23:43,378 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1674): Closed multi_test_table,ggg,1529623399386.10cdebbb720d98123807841a0febf405. 2018-06-21 23:23:43,379 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] handler.CloseRegionHandler(124): Closed multi_test_table,ggg,1529623399386.10cdebbb720d98123807841a0febf405. 2018-06-21 23:23:43,380 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=3.63 KB at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/40503e5e9a757a4ccf579bf29b8d78c6/.tmp/family/92ee034275a34f958eddcdbeb2c432a3 2018-06-21 23:23:43,382 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=4.10 KB at sequenceid=22 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/04d71925444c7c823938ba3d9ad0504d/.tmp/family/bc185397a8744389a43e344614f32323 2018-06-21 23:23:43,383 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 45cf8cdb5c344239afe0b620f5c1ebb6 2018-06-21 23:23:43,383 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=3.81 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/c793c2c3948aad84865d30ebf71a529e/.tmp/family/08f942bd57f04339b48c47335f2d395c 2018-06-21 23:23:43,385 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/dff6ab063ea78552b5e3c3ae7760ca5a/.tmp/family/45cf8cdb5c344239afe0b620f5c1ebb6 as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/dff6ab063ea78552b5e3c3ae7760ca5a/family/45cf8cdb5c344239afe0b620f5c1ebb6 2018-06-21 23:23:43,386 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2f9eee0859de402eb910b10632428c02 2018-06-21 23:23:43,387 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/7480a111a1f600aa34b97a77e47640bc/family/2f9eee0859de402eb910b10632428c02, entries=31, sequenceid=17, filesize=5.9 K 2018-06-21 23:23:43,391 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 92ee034275a34f958eddcdbeb2c432a3 2018-06-21 23:23:43,391 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 08f942bd57f04339b48c47335f2d395c 2018-06-21 23:23:43,392 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/c793c2c3948aad84865d30ebf71a529e/.tmp/family/08f942bd57f04339b48c47335f2d395c as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/c793c2c3948aad84865d30ebf71a529e/family/08f942bd57f04339b48c47335f2d395c 2018-06-21 23:23:43,392 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/40503e5e9a757a4ccf579bf29b8d78c6/.tmp/family/92ee034275a34f958eddcdbeb2c432a3 as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/40503e5e9a757a4ccf579bf29b8d78c6/family/92ee034275a34f958eddcdbeb2c432a3 2018-06-21 23:23:43,393 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for bc185397a8744389a43e344614f32323 2018-06-21 23:23:43,394 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/04d71925444c7c823938ba3d9ad0504d/.tmp/family/bc185397a8744389a43e344614f32323 as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/04d71925444c7c823938ba3d9ad0504d/family/bc185397a8744389a43e344614f32323 2018-06-21 23:23:43,401 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(2769): Finished flush of dataSize ~5.73 KB/5871, heapSize ~12.41 KB/12712, currentSize=0 B/0 for 7480a111a1f600aa34b97a77e47640bc in 127ms, sequenceid=17, compaction requested=false 2018-06-21 23:23:43,407 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for bc342c2523d744a58c14d1175d2fc967 2018-06-21 23:23:43,409 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/794028a2b8fc0a516890763306629d4c/.tmp/family/bc342c2523d744a58c14d1175d2fc967 as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/794028a2b8fc0a516890763306629d4c/family/bc342c2523d744a58c14d1175d2fc967 2018-06-21 23:23:43,412 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/7480a111a1f600aa34b97a77e47640bc/recovered.edits/20.seqid, newMaxSeqId=20, maxSeqId=1 2018-06-21 23:23:43,414 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 45cf8cdb5c344239afe0b620f5c1ebb6 2018-06-21 23:23:43,414 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for bc185397a8744389a43e344614f32323 2018-06-21 23:23:43,413 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 08f942bd57f04339b48c47335f2d395c 2018-06-21 23:23:43,415 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 92ee034275a34f958eddcdbeb2c432a3 2018-06-21 23:23:43,414 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/04d71925444c7c823938ba3d9ad0504d/family/bc185397a8744389a43e344614f32323, entries=30, sequenceid=22, filesize=5.9 K 2018-06-21 23:23:43,415 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1674): Closed multi_test_table,qqq,1529623399386.7480a111a1f600aa34b97a77e47640bc. 2018-06-21 23:23:43,414 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/dff6ab063ea78552b5e3c3ae7760ca5a/family/45cf8cdb5c344239afe0b620f5c1ebb6, entries=30, sequenceid=16, filesize=5.9 K 2018-06-21 23:23:43,415 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/40503e5e9a757a4ccf579bf29b8d78c6/family/92ee034275a34f958eddcdbeb2c432a3, entries=30, sequenceid=19, filesize=5.8 K 2018-06-21 23:23:43,415 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/c793c2c3948aad84865d30ebf71a529e/family/08f942bd57f04339b48c47335f2d395c, entries=33, sequenceid=18, filesize=6.0 K 2018-06-21 23:23:43,416 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] handler.CloseRegionHandler(124): Closed multi_test_table,qqq,1529623399386.7480a111a1f600aa34b97a77e47640bc. 2018-06-21 23:23:43,421 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(2769): Finished flush of dataSize ~4.10 KB/4194, heapSize ~9.02 KB/9240, currentSize=0 B/0 for 04d71925444c7c823938ba3d9ad0504d in 525ms, sequenceid=22, compaction requested=false 2018-06-21 23:23:43,421 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(2769): Finished flush of dataSize ~3.63 KB/3717, heapSize ~8.14 KB/8336, currentSize=0 B/0 for 40503e5e9a757a4ccf579bf29b8d78c6 in 527ms, sequenceid=19, compaction requested=false 2018-06-21 23:23:43,421 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(2769): Finished flush of dataSize ~5.70 KB/5832, heapSize ~12.30 KB/12600, currentSize=0 B/0 for dff6ab063ea78552b5e3c3ae7760ca5a in 136ms, sequenceid=16, compaction requested=false 2018-06-21 23:23:43,421 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(2769): Finished flush of dataSize ~3.81 KB/3903, heapSize ~8.68 KB/8888, currentSize=0 B/0 for c793c2c3948aad84865d30ebf71a529e in 525ms, sequenceid=18, compaction requested=false 2018-06-21 23:23:43,440 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for bc342c2523d744a58c14d1175d2fc967 2018-06-21 23:23:43,441 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/794028a2b8fc0a516890763306629d4c/family/bc342c2523d744a58c14d1175d2fc967, entries=18, sequenceid=17, filesize=5.4 K 2018-06-21 23:23:43,444 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(2769): Finished flush of dataSize ~1.79 KB/1836, heapSize ~6.02 KB/6168, currentSize=0 B/0 for 794028a2b8fc0a516890763306629d4c in 124ms, sequenceid=17, compaction requested=false 2018-06-21 23:23:43,444 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/c793c2c3948aad84865d30ebf71a529e/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=7 2018-06-21 23:23:43,457 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1674): Closed multi_test_table,eee,1529623399386.c793c2c3948aad84865d30ebf71a529e. 2018-06-21 23:23:43,457 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] handler.CloseRegionHandler(124): Closed multi_test_table,eee,1529623399386.c793c2c3948aad84865d30ebf71a529e. 2018-06-21 23:23:43,457 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/04d71925444c7c823938ba3d9ad0504d/recovered.edits/25.seqid, newMaxSeqId=25, maxSeqId=10 2018-06-21 23:23:43,458 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/40503e5e9a757a4ccf579bf29b8d78c6/recovered.edits/22.seqid, newMaxSeqId=22, maxSeqId=7 2018-06-21 23:23:43,459 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1674): Closed multi_test_table,iii,1529623399386.04d71925444c7c823938ba3d9ad0504d. 2018-06-21 23:23:43,459 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] handler.CloseRegionHandler(124): Closed multi_test_table,iii,1529623399386.04d71925444c7c823938ba3d9ad0504d. 2018-06-21 23:23:43,459 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1674): Closed multi_test_table,uuu,1529623399386.40503e5e9a757a4ccf579bf29b8d78c6. 2018-06-21 23:23:43,459 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1520): Closing e66f718b26d1c4420497ccfb42f08321, disabling compactions & flushes 2018-06-21 23:23:43,460 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1560): Updates disabled for region multi_test_table,ddd,1529623399386.e66f718b26d1c4420497ccfb42f08321. 2018-06-21 23:23:43,459 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] handler.CloseRegionHandler(124): Closed multi_test_table,uuu,1529623399386.40503e5e9a757a4ccf579bf29b8d78c6. 2018-06-21 23:23:43,460 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/dff6ab063ea78552b5e3c3ae7760ca5a/recovered.edits/19.seqid, newMaxSeqId=19, maxSeqId=1 2018-06-21 23:23:43,460 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=6.10 KB heapSize=13.28 KB 2018-06-21 23:23:43,460 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1520): Closing 124da0ae361b264a91ea989bcd12b459, disabling compactions & flushes 2018-06-21 23:23:43,460 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1560): Updates disabled for region multi_test_table,ccc,1529623399386.124da0ae361b264a91ea989bcd12b459. 2018-06-21 23:23:43,461 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=6.47 KB heapSize=14.47 KB 2018-06-21 23:23:43,461 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1674): Closed multi_test_table,ttt,1529623399386.dff6ab063ea78552b5e3c3ae7760ca5a. 2018-06-21 23:23:43,462 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] handler.CloseRegionHandler(124): Closed multi_test_table,ttt,1529623399386.dff6ab063ea78552b5e3c3ae7760ca5a. 2018-06-21 23:23:43,462 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1520): Closing fee67ae5d4f2b2be35fa516877a8d173, disabling compactions & flushes 2018-06-21 23:23:43,462 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1560): Updates disabled for region multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173. 2018-06-21 23:23:43,462 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(2560): Flushing 1/1 column families, dataSize=3.61 KB heapSize=7.63 KB 2018-06-21 23:23:43,467 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/794028a2b8fc0a516890763306629d4c/recovered.edits/20.seqid, newMaxSeqId=20, maxSeqId=5 2018-06-21 23:23:43,469 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1674): Closed multi_test_table,zzz,1529623399386.794028a2b8fc0a516890763306629d4c. 2018-06-21 23:23:43,469 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] handler.CloseRegionHandler(124): Closed multi_test_table,zzz,1529623399386.794028a2b8fc0a516890763306629d4c. 2018-06-21 23:23:43,486 INFO [RS:4;asf911:58235] regionserver.HRegionServer(1097): stopping server asf911.gq1.ygridcore.net,58235,1529623390382; all regions closed. 2018-06-21 23:23:43,494 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(874): complete file /user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,58235,1529623390382/asf911.gq1.ygridcore.net%2C58235%2C1529623390382.1529623393801 not finished, retry = 0 2018-06-21 23:23:43,495 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741832_1008{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW]]} size 62449 2018-06-21 23:23:43,495 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741832_1008 size 62449 2018-06-21 23:23:43,495 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741832_1008 size 62449 2018-06-21 23:23:43,500 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741928_1104{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|FINALIZED]]} size 0 2018-06-21 23:23:43,500 INFO [RS:3;asf911:56613] regionserver.HRegionServer(1097): stopping server asf911.gq1.ygridcore.net,56613,1529623390297; all regions closed. 2018-06-21 23:23:43,500 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741928_1104{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|FINALIZED]]} size 0 2018-06-21 23:23:43,501 INFO [RS:5;asf911:34033] regionserver.HRegionServer(1097): stopping server asf911.gq1.ygridcore.net,34033,1529623407773; all regions closed. 2018-06-21 23:23:43,501 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741927_1103{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW]]} size 0 2018-06-21 23:23:43,501 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741928_1104{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|FINALIZED]]} size 0 2018-06-21 23:23:43,503 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741927_1103{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW]]} size 0 2018-06-21 23:23:43,503 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=6.47 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/124da0ae361b264a91ea989bcd12b459/.tmp/family/1b9c91b2e17040d7bad2ee31ba3a1374 2018-06-21 23:23:43,509 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741927_1103{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW]]} size 0 2018-06-21 23:23:43,520 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(874): complete file /user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,34033,1529623407773/asf911.gq1.ygridcore.net%2C34033%2C1529623407773.meta.1529623421561.meta not finished, retry = 0 2018-06-21 23:23:43,520 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741902_1078{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW]]} size 653 2018-06-21 23:23:43,523 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=3.61 KB at sequenceid=25 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/.tmp/family/19af3abd1e0540a8bd39bc489bd29a8b 2018-06-21 23:23:43,523 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741902_1078 size 653 2018-06-21 23:23:43,523 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741902_1078 size 653 2018-06-21 23:23:43,523 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741830_1006{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:43,524 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741830_1006{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:43,524 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741830_1006{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:43,525 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741929_1105{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:43,526 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741929_1105{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:43,526 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741929_1105{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:43,528 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=6.10 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/e66f718b26d1c4420497ccfb42f08321/.tmp/family/35b70a4808df49c09d4a20ca1961ad00 2018-06-21 23:23:43,532 DEBUG [RS:3;asf911:56613] wal.AbstractFSWAL(854): Moved 1 WAL file(s) to /user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/oldWALs 2018-06-21 23:23:43,532 INFO [RS:3;asf911:56613] wal.AbstractFSWAL(857): Closed WAL: AsyncFSWAL asf911.gq1.ygridcore.net%2C56613%2C1529623390297:(num 1529623393801) 2018-06-21 23:23:43,532 DEBUG [RS:3;asf911:56613] ipc.AbstractRpcClient(483): Stopping rpc client 2018-06-21 23:23:43,532 INFO [RS:3;asf911:56613] regionserver.Leases(149): Closed leases 2018-06-21 23:23:43,533 INFO [RS:3;asf911:56613] hbase.ChoreService(327): Chore service for: regionserver/asf911:0 had [[ScheduledChore: Name: MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region asf911.gq1.ygridcore.net,56613,1529623390297 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS]] on shutdown 2018-06-21 23:23:43,533 INFO [RS:3;asf911:56613] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-06-21 23:23:43,533 INFO [RS:3;asf911:56613] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-06-21 23:23:43,533 INFO [RS:3;asf911:56613] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-06-21 23:23:43,533 INFO [regionserver/asf911:0.logRoller] regionserver.LogRoller(212): LogRoller exiting. 2018-06-21 23:23:43,536 INFO [RS:3;asf911:56613] ipc.NettyRpcServer(143): Stopping server on /67.195.81.155:56613 2018-06-21 23:23:43,542 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 1b9c91b2e17040d7bad2ee31ba3a1374 2018-06-21 23:23:43,543 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 19af3abd1e0540a8bd39bc489bd29a8b 2018-06-21 23:23:43,543 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 35b70a4808df49c09d4a20ca1961ad00 2018-06-21 23:23:43,544 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/124da0ae361b264a91ea989bcd12b459/.tmp/family/1b9c91b2e17040d7bad2ee31ba3a1374 as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/124da0ae361b264a91ea989bcd12b459/family/1b9c91b2e17040d7bad2ee31ba3a1374 2018-06-21 23:23:43,545 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/e66f718b26d1c4420497ccfb42f08321/.tmp/family/35b70a4808df49c09d4a20ca1961ad00 as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/e66f718b26d1c4420497ccfb42f08321/family/35b70a4808df49c09d4a20ca1961ad00 2018-06-21 23:23:43,545 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/.tmp/family/19af3abd1e0540a8bd39bc489bd29a8b as hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/family/19af3abd1e0540a8bd39bc489bd29a8b 2018-06-21 23:23:43,549 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:43,549 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:43,549 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:43,550 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:43,550 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:56613-0x16424a89dd40004, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 2018-06-21 23:23:43,555 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 1b9c91b2e17040d7bad2ee31ba3a1374 2018-06-21 23:23:43,555 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 35b70a4808df49c09d4a20ca1961ad00 2018-06-21 23:23:43,555 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/124da0ae361b264a91ea989bcd12b459/family/1b9c91b2e17040d7bad2ee31ba3a1374, entries=36, sequenceid=16, filesize=6.0 K 2018-06-21 23:23:43,555 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.StoreFileReader(496): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 19af3abd1e0540a8bd39bc489bd29a8b 2018-06-21 23:23:43,555 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/e66f718b26d1c4420497ccfb42f08321/family/35b70a4808df49c09d4a20ca1961ad00, entries=33, sequenceid=16, filesize=6.0 K 2018-06-21 23:23:43,555 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HStore(1070): Added hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/family/19af3abd1e0540a8bd39bc489bd29a8b, entries=31, sequenceid=25, filesize=5.9 K 2018-06-21 23:23:43,558 INFO [RS:3;asf911:56613] regionserver.HRegionServer(1153): Exiting; stopping=asf911.gq1.ygridcore.net,56613,1529623390297; zookeeper connection closed. 2018-06-21 23:23:43,558 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:43,559 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@2c1a7573] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(221): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@2c1a7573 2018-06-21 23:23:43,559 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:43,559 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(2769): Finished flush of dataSize ~6.47 KB/6621, heapSize ~14.68 KB/15032, currentSize=0 B/0 for 124da0ae361b264a91ea989bcd12b459 in 99ms, sequenceid=16, compaction requested=false 2018-06-21 23:23:43,559 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:43,559 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(2769): Finished flush of dataSize ~3.61 KB/3692, heapSize ~8.05 KB/8240, currentSize=0 B/0 for fee67ae5d4f2b2be35fa516877a8d173 in 97ms, sequenceid=25, compaction requested=true 2018-06-21 23:23:43,559 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(2769): Finished flush of dataSize ~6.10 KB/6243, heapSize ~13.49 KB/13816, currentSize=0 B/0 for e66f718b26d1c4420497ccfb42f08321 in 99ms, sequenceid=16, compaction requested=false 2018-06-21 23:23:43,561 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:43,561 INFO [Time-limited test-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(122): /hbase/rs/asf911.gq1.ygridcore.net,56613,1529623390297 znode expired, triggering replicatorRemoved event 2018-06-21 23:23:43,561 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:43,562 INFO [RegionServerTracker-0] master.RegionServerTracker(159): RegionServer ephemeral node deleted, processing expiration [asf911.gq1.ygridcore.net,56613,1529623390297] 2018-06-21 23:23:43,562 INFO [RegionServerTracker-0] master.ServerManager(597): Cluster shutdown set; asf911.gq1.ygridcore.net,56613,1529623390297 expired; onlineServers=4 2018-06-21 23:23:43,562 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:43,563 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:43,563 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:43,564 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:43,567 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/fee67ae5d4f2b2be35fa516877a8d173/recovered.edits/28.seqid, newMaxSeqId=28, maxSeqId=13 2018-06-21 23:23:43,568 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/e66f718b26d1c4420497ccfb42f08321/recovered.edits/19.seqid, newMaxSeqId=19, maxSeqId=1 2018-06-21 23:23:43,568 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] wal.WALSplitter(678): Wrote file=hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/data/default/multi_test_table/124da0ae361b264a91ea989bcd12b459/recovered.edits/19.seqid, newMaxSeqId=19, maxSeqId=1 2018-06-21 23:23:43,568 INFO [RS_CLOSE_REGION-regionserver/asf911:0-2] regionserver.HRegion(1674): Closed multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173. 2018-06-21 23:23:43,568 INFO [RS_CLOSE_REGION-regionserver/asf911:0-0] regionserver.HRegion(1674): Closed multi_test_table,ddd,1529623399386.e66f718b26d1c4420497ccfb42f08321. 2018-06-21 23:23:43,568 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-2] handler.CloseRegionHandler(124): Closed multi_test_table,kkk,1529623399386.fee67ae5d4f2b2be35fa516877a8d173. 2018-06-21 23:23:43,569 INFO [RS_CLOSE_REGION-regionserver/asf911:0-1] regionserver.HRegion(1674): Closed multi_test_table,ccc,1529623399386.124da0ae361b264a91ea989bcd12b459. 2018-06-21 23:23:43,569 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-1] handler.CloseRegionHandler(124): Closed multi_test_table,ccc,1529623399386.124da0ae361b264a91ea989bcd12b459. 2018-06-21 23:23:43,569 DEBUG [RS_CLOSE_REGION-regionserver/asf911:0-0] handler.CloseRegionHandler(124): Closed multi_test_table,ddd,1529623399386.e66f718b26d1c4420497ccfb42f08321. 2018-06-21 23:23:43,600 DEBUG [RS:4;asf911:58235] wal.AbstractFSWAL(854): Moved 1 WAL file(s) to /user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/oldWALs 2018-06-21 23:23:43,600 INFO [RS:4;asf911:58235] wal.AbstractFSWAL(857): Closed WAL: AsyncFSWAL asf911.gq1.ygridcore.net%2C58235%2C1529623390382:(num 1529623393801) 2018-06-21 23:23:43,601 DEBUG [RS:4;asf911:58235] ipc.AbstractRpcClient(483): Stopping rpc client 2018-06-21 23:23:43,601 INFO [RS:4;asf911:58235] regionserver.Leases(149): Closed leases 2018-06-21 23:23:43,601 INFO [regionserver/asf911:0.Chore.1] hbase.ScheduledChore(180): Chore: MemstoreFlusherChore was stopped 2018-06-21 23:23:43,601 INFO [RS:4;asf911:58235] hbase.ChoreService(327): Chore service for: regionserver/asf911:0 had [[ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region asf911.gq1.ygridcore.net,58235,1529623390382 Period: 120000 Unit: MILLISECONDS]] on shutdown 2018-06-21 23:23:43,601 INFO [RS:4;asf911:58235] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-06-21 23:23:43,601 INFO [RS:4;asf911:58235] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-06-21 23:23:43,601 INFO [RS:4;asf911:58235] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-06-21 23:23:43,601 INFO [regionserver/asf911:0.logRoller] regionserver.LogRoller(212): LogRoller exiting. 2018-06-21 23:23:43,604 INFO [RS:4;asf911:58235] ipc.NettyRpcServer(143): Stopping server on /67.195.81.155:58235 2018-06-21 23:23:43,610 INFO [regionserver/asf911:0.leaseChecker] regionserver.Leases(149): Closed leases 2018-06-21 23:23:43,610 INFO [regionserver/asf911:0.leaseChecker] regionserver.Leases(149): Closed leases 2018-06-21 23:23:43,611 INFO [regionserver/asf911:0.leaseChecker] regionserver.Leases(149): Closed leases 2018-06-21 23:23:43,616 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:58235-0x16424a89dd40005, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:43,616 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:43,616 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:43,616 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 2018-06-21 23:23:43,624 INFO [RS:4;asf911:58235] regionserver.HRegionServer(1153): Exiting; stopping=asf911.gq1.ygridcore.net,58235,1529623390382; zookeeper connection closed. 2018-06-21 23:23:43,625 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@787e6d1b] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(221): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@787e6d1b 2018-06-21 23:23:43,625 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:43,625 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:43,626 DEBUG [RS:5;asf911:34033] wal.AbstractFSWAL(854): Moved 1 WAL file(s) to /user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/oldWALs 2018-06-21 23:23:43,626 INFO [RS:5;asf911:34033] wal.AbstractFSWAL(857): Closed WAL: AsyncFSWAL asf911.gq1.ygridcore.net%2C34033%2C1529623407773.meta:.meta(num 1529623421561) 2018-06-21 23:23:43,626 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:43,626 INFO [Time-limited test-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(122): /hbase/rs/asf911.gq1.ygridcore.net,58235,1529623390382 znode expired, triggering replicatorRemoved event 2018-06-21 23:23:43,626 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:43,627 INFO [RegionServerTracker-0] master.RegionServerTracker(159): RegionServer ephemeral node deleted, processing expiration [asf911.gq1.ygridcore.net,58235,1529623390382] 2018-06-21 23:23:43,627 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:43,627 INFO [RegionServerTracker-0] master.ServerManager(597): Cluster shutdown set; asf911.gq1.ygridcore.net,58235,1529623390382 expired; onlineServers=3 2018-06-21 23:23:43,633 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:43,633 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(874): complete file /user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,34033,1529623407773/asf911.gq1.ygridcore.net%2C34033%2C1529623407773.1529623408969 not finished, retry = 0 2018-06-21 23:23:43,634 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:43,634 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741879_1055{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW]]} size 6122 2018-06-21 23:23:43,634 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741879_1055 size 6122 2018-06-21 23:23:43,634 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741879_1055 size 6122 2018-06-21 23:23:43,636 INFO [regionserver/asf911:0.Chore.1] hbase.ScheduledChore(180): Chore: MemstoreFlusherChore was stopped 2018-06-21 23:23:43,701 INFO [RS:2;asf911:52507] regionserver.HRegionServer(1097): stopping server asf911.gq1.ygridcore.net,52507,1529623390228; all regions closed. 2018-06-21 23:23:43,708 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741834_1010{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-189d75fd-2cba-4781-8f7e-7f1fecbda947:NORMAL:127.0.0.1:33402|RBW]]} size 58880 2018-06-21 23:23:43,708 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(874): complete file /user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/WALs/asf911.gq1.ygridcore.net,52507,1529623390228/asf911.gq1.ygridcore.net%2C52507%2C1529623390228.1529623393801 not finished, retry = 0 2018-06-21 23:23:43,708 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741834_1010 size 58880 2018-06-21 23:23:43,708 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741834_1010 size 58880 2018-06-21 23:23:43,715 INFO [M:0;asf911:55146] regionserver.HRegionServer(2148): ***** STOPPING region server 'asf911.gq1.ygridcore.net,55146,1529623388534' ***** 2018-06-21 23:23:43,715 INFO [M:0;asf911:55146] regionserver.HRegionServer(2162): STOPPED: Exiting; cluster shutdown set and not carrying any regions 2018-06-21 23:23:43,724 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-06-21 23:23:43,724 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-06-21 23:23:43,724 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-06-21 23:23:43,725 INFO [M:0;asf911:55146] regionserver.SplitLogWorker(166): Sending interrupt to stop the worker thread 2018-06-21 23:23:43,725 INFO [M:0;asf911:55146] regionserver.HeapMemoryManager(221): Stopping 2018-06-21 23:23:43,725 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(357): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-06-21 23:23:43,725 INFO [SplitLogWorker-asf911:55146] regionserver.SplitLogWorker(148): SplitLogWorker interrupted. Exiting. 2018-06-21 23:23:43,725 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(357): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-06-21 23:23:43,727 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(383): MemStoreFlusher.1 exiting 2018-06-21 23:23:43,725 INFO [SplitLogWorker-asf911:55146] regionserver.SplitLogWorker(157): SplitLogWorker asf911.gq1.ygridcore.net,55146,1529623388534 exiting 2018-06-21 23:23:43,725 INFO [M:0;asf911:55146] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager gracefully. 2018-06-21 23:23:43,725 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(383): MemStoreFlusher.0 exiting 2018-06-21 23:23:43,725 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(357): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-06-21 23:23:43,727 INFO [M:0;asf911:55146] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2018-06-21 23:23:43,727 INFO [M:0;asf911:55146] regionserver.HRegionServer(1069): stopping server asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:43,728 DEBUG [M:0;asf911:55146] zookeeper.MetaTableLocator(642): Stopping MetaTableLocator 2018-06-21 23:23:43,728 INFO [M:0;asf911:55146] regionserver.HRegionServer(1097): stopping server asf911.gq1.ygridcore.net,55146,1529623388534; all regions closed. 2018-06-21 23:23:43,733 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741883_1059{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:43,733 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741883_1059{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:43,733 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741883_1059{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|RBW]]} size 0 2018-06-21 23:23:43,737 DEBUG [M:0;asf911:55146] wal.AbstractFSWAL(854): Moved 1 WAL file(s) to /user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/oldWALs 2018-06-21 23:23:43,737 INFO [M:0;asf911:55146] wal.AbstractFSWAL(857): Closed WAL: AsyncFSWAL asf911.gq1.ygridcore.net%2C55146%2C1529623388534.meta:.meta(num 1529623415657) 2018-06-21 23:23:43,737 DEBUG [RS:5;asf911:34033] wal.AbstractFSWAL(854): Moved 1 WAL file(s) to /user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/oldWALs 2018-06-21 23:23:43,737 INFO [RS:5;asf911:34033] wal.AbstractFSWAL(857): Closed WAL: AsyncFSWAL asf911.gq1.ygridcore.net%2C34033%2C1529623407773:(num 1529623408969) 2018-06-21 23:23:43,737 DEBUG [RS:5;asf911:34033] ipc.AbstractRpcClient(483): Stopping rpc client 2018-06-21 23:23:43,738 INFO [RS:5;asf911:34033] regionserver.Leases(149): Closed leases 2018-06-21 23:23:43,738 INFO [RS:5;asf911:34033] hbase.ChoreService(327): Chore service for: regionserver/asf911:0 had [[ScheduledChore: Name: MovedRegionsCleaner for region asf911.gq1.ygridcore.net,34033,1529623407773 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS]] on shutdown 2018-06-21 23:23:43,738 INFO [regionserver/asf911:0.logRoller] regionserver.LogRoller(212): LogRoller exiting. 2018-06-21 23:23:43,741 INFO [RS:5;asf911:34033] ipc.NettyRpcServer(143): Stopping server on /67.195.81.155:34033 2018-06-21 23:23:43,748 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741831_1007{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:43,749 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:33402 is added to blk_1073741831_1007{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:43,749 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741831_1007{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2bcd7c05-54b4-4126-9be1-2aaa6a858f0d:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-b2ed598d-9dcf-4db9-a4a0-d09e192dac4e:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-d4d0c936-07f4-4962-b708-add7d68719f3:NORMAL:127.0.0.1:33402|RBW]]} size 0 2018-06-21 23:23:43,749 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:43,749 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:34033-0x16424a89dd4000f, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:43,749 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-06-21 23:23:43,749 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,34033,1529623407773 2018-06-21 23:23:43,752 DEBUG [M:0;asf911:55146] wal.AbstractFSWAL(854): Moved 1 WAL file(s) to /user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/oldWALs 2018-06-21 23:23:43,752 INFO [M:0;asf911:55146] wal.AbstractFSWAL(857): Closed WAL: AsyncFSWAL asf911.gq1.ygridcore.net%2C55146%2C1529623388534:(num 1529623393801) 2018-06-21 23:23:43,752 DEBUG [M:0;asf911:55146] ipc.AbstractRpcClient(483): Stopping rpc client 2018-06-21 23:23:43,753 INFO [M:0;asf911:55146] regionserver.Leases(149): Closed leases 2018-06-21 23:23:43,753 INFO [M:0;asf911:55146] hbase.ChoreService(327): Chore service for: master/asf911:0 had [[ScheduledChore: Name: asf911.gq1.ygridcore.net,55146,1529623388534-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,55146,1529623388534-RegionNormalizerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: ReplicationBarrierCleaner Period: 43200000 Unit: MILLISECONDS], [ScheduledChore: Name: MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS], [ScheduledChore: Name: CatalogJanitor-asf911:55146 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,55146,1529623388534-ExpiredMobFileCleanerChore Period: 86400 Unit: SECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,55146,1529623388534-MobCompactionChore Period: 604800 Unit: SECONDS], [ScheduledChore: Name: FlushedSequenceIdFlusher Period: 10800000 Unit: MILLISECONDS], [ScheduledChore: Name: asf911.gq1.ygridcore.net,55146,1529623388534-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: LogsCleaner Period: 600000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region asf911.gq1.ygridcore.net,55146,1529623388534 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: HFileCleaner Period: 600000 Unit: MILLISECONDS]] on shutdown 2018-06-21 23:23:43,753 INFO [M:0;asf911:55146] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-06-21 23:23:43,753 INFO [M:0;asf911:55146] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-06-21 23:23:43,753 INFO [M:0;asf911:55146] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-06-21 23:23:43,753 INFO [master/asf911:0.logRoller] regionserver.LogRoller(212): LogRoller exiting. 2018-06-21 23:23:43,754 INFO [M:0;asf911:55146] master.MasterMobCompactionThread(175): Waiting for Mob Compaction Thread to finish... 2018-06-21 23:23:43,754 INFO [M:0;asf911:55146] master.MasterMobCompactionThread(175): Waiting for Region Server Mob Compaction Thread to finish... 2018-06-21 23:23:43,755 DEBUG [M:0;asf911:55146] master.HMaster(1284): Stopping service threads 2018-06-21 23:23:43,755 DEBUG [OldWALsCleaner-1] cleaner.LogCleaner(176): Exiting cleaner. 2018-06-21 23:23:43,755 DEBUG [Thread-229-HFileCleaner.large.0-1529623392261] cleaner.HFileCleaner(253): Exit Thread[Thread-229-HFileCleaner.large.0-1529623392261,5,FailOnTimeoutGroup] 2018-06-21 23:23:43,755 DEBUG [OldWALsCleaner-0] cleaner.LogCleaner(176): Exiting cleaner. 2018-06-21 23:23:43,755 DEBUG [Thread-229-HFileCleaner.small.0-1529623392262] cleaner.HFileCleaner(253): Exit Thread[Thread-229-HFileCleaner.small.0-1529623392262,5,FailOnTimeoutGroup] 2018-06-21 23:23:43,758 INFO [RS:5;asf911:34033] regionserver.HRegionServer(1153): Exiting; stopping=asf911.gq1.ygridcore.net,34033,1529623407773; zookeeper connection closed. 2018-06-21 23:23:43,758 DEBUG [M:0;asf911:55146] zookeeper.ZKUtil(614): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2018-06-21 23:23:43,758 WARN [M:0;asf911:55146] master.ActiveMasterManager(271): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2018-06-21 23:23:43,758 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@1d3f45a7] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(221): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@1d3f45a7 2018-06-21 23:23:43,759 INFO [M:0;asf911:55146] master.ServerManager(1056): Writing .lastflushedseqids file at: hdfs://localhost:56139/user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/.lastflushedseqids 2018-06-21 23:23:43,780 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741930_1106{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-75ab1804-842f-457d-9d47-0b2ec0fd81e4:NORMAL:127.0.0.1:53553|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW]]} size 0 2018-06-21 23:23:43,780 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:53553 is added to blk_1073741930_1106{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|FINALIZED]]} size 0 2018-06-21 23:23:43,780 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741930_1106{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-615ebfc4-105b-4470-b106-59b6b4ae84d8:NORMAL:127.0.0.1:53553|FINALIZED], ReplicaUC[[DISK]DS-85f0e371-3357-43c0-95ea-ca66d237b4a8:NORMAL:127.0.0.1:42074|FINALIZED]]} size 0 2018-06-21 23:23:43,781 INFO [M:0;asf911:55146] assignment.AssignmentManager(234): Stopping assignment manager 2018-06-21 23:23:43,783 INFO [M:0;asf911:55146] procedure2.RemoteProcedureDispatcher(116): Stopping procedure remote dispatcher 2018-06-21 23:23:43,786 INFO [M:0;asf911:55146] wal.WALProcedureStore(290): Stopping the WAL Procedure Store, isAbort=false 2018-06-21 23:23:43,800 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57095 is added to blk_1073741829_1005{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW]]} size 202 2018-06-21 23:23:43,802 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54793 is added to blk_1073741829_1005{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW]]} size 202 2018-06-21 23:23:43,802 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:42074 is added to blk_1073741829_1005{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ea4461c1-0232-45b6-ab90-96c5b7ff2f26:NORMAL:127.0.0.1:54793|RBW], ReplicaUC[[DISK]DS-901750e1-a133-4f10-85af-4c0c301de1f4:NORMAL:127.0.0.1:57095|RBW], ReplicaUC[[DISK]DS-7181f43d-72ad-44c3-b95b-62ec3d72c2a7:NORMAL:127.0.0.1:42074|RBW]]} size 202 2018-06-21 23:23:43,803 INFO [M:0;asf911:55146] hbase.ChoreService(327): Chore service for: master/asf911:0.splitLogManager. had [[ScheduledChore: Name: SplitLogManager Timeout Monitor Period: 1000 Unit: MILLISECONDS]] on shutdown 2018-06-21 23:23:43,803 INFO [M:0;asf911:55146] flush.MasterFlushTableProcedureManager(81): stop: server shutting down. 2018-06-21 23:23:43,804 INFO [M:0;asf911:55146] ipc.NettyRpcServer(143): Stopping server on /67.195.81.155:55146 2018-06-21 23:23:43,812 DEBUG [RS:2;asf911:52507] wal.AbstractFSWAL(854): Moved 1 WAL file(s) to /user/jenkins/test-data/d3a75823-c9c5-4206-b9ab-6ea68f489fba/oldWALs 2018-06-21 23:23:43,812 INFO [RS:2;asf911:52507] wal.AbstractFSWAL(857): Closed WAL: AsyncFSWAL asf911.gq1.ygridcore.net%2C52507%2C1529623390228:(num 1529623393801) 2018-06-21 23:23:43,812 DEBUG [RS:2;asf911:52507] ipc.AbstractRpcClient(483): Stopping rpc client 2018-06-21 23:23:43,812 INFO [RS:2;asf911:52507] regionserver.Leases(149): Closed leases 2018-06-21 23:23:43,813 INFO [RS:2;asf911:52507] hbase.ChoreService(327): Chore service for: regionserver/asf911:0 had [[ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region asf911.gq1.ygridcore.net,52507,1529623390228 Period: 120000 Unit: MILLISECONDS]] on shutdown 2018-06-21 23:23:43,813 INFO [RS:2;asf911:52507] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-06-21 23:23:43,813 INFO [regionserver/asf911:0.logRoller] regionserver.LogRoller(212): LogRoller exiting. 2018-06-21 23:23:43,813 INFO [RS:2;asf911:52507] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-06-21 23:23:43,813 INFO [RS:2;asf911:52507] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-06-21 23:23:43,815 INFO [RS:2;asf911:52507] ipc.NettyRpcServer(143): Stopping server on /67.195.81.155:52507 2018-06-21 23:23:43,817 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:55146-0x16424a89dd40000, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:43,818 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,55146,1529623388534 2018-06-21 23:23:43,824 INFO [M:0;asf911:55146] regionserver.HRegionServer(1153): Exiting; stopping=asf911.gq1.ygridcore.net,55146,1529623388534; zookeeper connection closed. 2018-06-21 23:23:43,833 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:52507-0x16424a89dd40003, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf911.gq1.ygridcore.net,52507,1529623390228 2018-06-21 23:23:43,841 INFO [RS:2;asf911:52507] regionserver.HRegionServer(1153): Exiting; stopping=asf911.gq1.ygridcore.net,52507,1529623390228; zookeeper connection closed. 2018-06-21 23:23:43,841 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@591fcbf1] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(221): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@591fcbf1 2018-06-21 23:23:43,842 INFO [Time-limited test] util.JVMClusterUtil(326): Shutdown of 1 master(s) and 6 regionserver(s) complete 2018-06-21 23:23:43,842 WARN [Time-limited test] datanode.DirectoryScanner(529): DirectoryScanner: shutdown has been called 2018-06-21 23:23:43,863 INFO [Time-limited test] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2018-06-21 23:23:43,973 WARN [DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data9/, [DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data10/]] heartbeating to localhost/127.0.0.1:56139] datanode.IncrementalBlockReportManager(132): IncrementalBlockReportManager interrupted 2018-06-21 23:23:43,974 WARN [DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data9/, [DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data10/]] heartbeating to localhost/127.0.0.1:56139] datanode.BPServiceActor(670): Ending block pool service for: Block pool BP-1812449274-67.195.81.155-1529623383006 (Datanode Uuid 24acd91f-11d1-457c-a0e1-2b33943135cf) service to localhost/127.0.0.1:56139 2018-06-21 23:23:43,975 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@69ed135] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:57095 to delete [blk_1073741890_1066, blk_1073741891_1067, blk_1073741892_1068, blk_1073741886_1062, blk_1073741887_1063] 2018-06-21 23:23:43,975 INFO [org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor@69ed135] blockmanagement.BlockManager(3483): BLOCK* BlockManager: ask 127.0.0.1:42074 to delete [blk_1073741888_1064, blk_1073741889_1065, blk_1073741892_1068, blk_1073741886_1062, blk_1073741887_1063] 2018-06-21 23:23:43,980 WARN [Time-limited test] datanode.DirectoryScanner(529): DirectoryScanner: shutdown has been called 2018-06-21 23:23:43,992 INFO [Time-limited test] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2018-06-21 23:23:44,101 WARN [DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data7/, [DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data8/]] heartbeating to localhost/127.0.0.1:56139] datanode.IncrementalBlockReportManager(132): IncrementalBlockReportManager interrupted 2018-06-21 23:23:44,101 WARN [DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data7/, [DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data8/]] heartbeating to localhost/127.0.0.1:56139] datanode.BPServiceActor(670): Ending block pool service for: Block pool BP-1812449274-67.195.81.155-1529623383006 (Datanode Uuid 61f88414-0be1-4db2-bf5a-0475ff8fcc50) service to localhost/127.0.0.1:56139 2018-06-21 23:23:44,104 WARN [Time-limited test] datanode.DirectoryScanner(529): DirectoryScanner: shutdown has been called 2018-06-21 23:23:44,116 INFO [Time-limited test] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2018-06-21 23:23:44,225 WARN [DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data5/, [DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data6/]] heartbeating to localhost/127.0.0.1:56139] datanode.IncrementalBlockReportManager(132): IncrementalBlockReportManager interrupted 2018-06-21 23:23:44,225 WARN [DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data5/, [DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data6/]] heartbeating to localhost/127.0.0.1:56139] datanode.BPServiceActor(670): Ending block pool service for: Block pool BP-1812449274-67.195.81.155-1529623383006 (Datanode Uuid 937d2ef4-9d52-43ce-9fb6-245f9750c658) service to localhost/127.0.0.1:56139 2018-06-21 23:23:44,232 WARN [Time-limited test] datanode.DirectoryScanner(529): DirectoryScanner: shutdown has been called 2018-06-21 23:23:44,246 INFO [Time-limited test] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2018-06-21 23:23:44,355 WARN [DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data3/, [DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data4/]] heartbeating to localhost/127.0.0.1:56139] datanode.IncrementalBlockReportManager(132): IncrementalBlockReportManager interrupted 2018-06-21 23:23:44,355 WARN [DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data3/, [DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data4/]] heartbeating to localhost/127.0.0.1:56139] datanode.BPServiceActor(670): Ending block pool service for: Block pool BP-1812449274-67.195.81.155-1529623383006 (Datanode Uuid 6ec31509-2550-4a57-907a-3986dd3cfb9c) service to localhost/127.0.0.1:56139 2018-06-21 23:23:44,361 WARN [Time-limited test] datanode.DirectoryScanner(529): DirectoryScanner: shutdown has been called 2018-06-21 23:23:44,376 INFO [Time-limited test] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2018-06-21 23:23:44,595 WARN [DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:56139] datanode.IncrementalBlockReportManager(132): IncrementalBlockReportManager interrupted 2018-06-21 23:23:44,599 WARN [DataNode: [[[DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data1/, [DISK]file:/home/jenkins/jenkins-slave/workspace/HBase-Flaky-Tests/hbase-server/target/test-data/7e66c67d-a23e-408c-a385-9cf16a0de0f1/cluster_0b3169b0-b07d-4eb9-af38-56e627e9fcc0/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:56139] datanode.BPServiceActor(670): Ending block pool service for: Block pool BP-1812449274-67.195.81.155-1529623383006 (Datanode Uuid c317d2ee-b8dd-497d-a934-c9aeee7ee76e) service to localhost/127.0.0.1:56139 2018-06-21 23:23:44,600 INFO [Finalizer] zookeeper.ReadOnlyZKClient(350): Close zookeeper connection 0x66c2cb03 to localhost:58989 2018-06-21 23:23:44,600 DEBUG [Finalizer] ipc.AbstractRpcClient(483): Stopping rpc client 2018-06-21 23:23:44,627 INFO [Time-limited test] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2018-06-21 23:23:44,716 INFO [master/asf911:0.leaseChecker] regionserver.Leases(149): Closed leases 2018-06-21 23:23:44,732 ERROR [Time-limited test] server.ZooKeeperServer(472): ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes 2018-06-21 23:23:44,737 INFO [Time-limited test] zookeeper.MiniZooKeeperCluster(324): Shutdown MiniZK cluster with all ZK servers 2018-06-21 23:23:44,784 INFO [Time-limited test] hbase.HBaseTestingUtility(1104): Minicluster is down 2018-06-21 23:23:44,833 DEBUG [Thread-229-EventThread] zookeeper.ZKWatcher(478): replicationLogCleaner-0x16424a89dd4000c, quorum=localhost:58989, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2018-06-21 23:23:44,833 DEBUG [Thread-229-EventThread] zookeeper.ZKWatcher(548): replicationLogCleaner-0x16424a89dd4000c, quorum=localhost:58989, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2018-06-21 23:23:44,875 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@30f0c6fa 2018-06-21 23:23:44,875 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. 2018-06-21 23:23:44,875 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@30f0c6fa 2018-06-21 23:23:44,875 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. 2018-06-21 23:23:44,876 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@30f0c6fa 2018-06-21 23:23:44,876 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. 2018-06-21 23:23:44,876 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@30f0c6fa 2018-06-21 23:23:44,876 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. 2018-06-21 23:23:44,876 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@30f0c6fa 2018-06-21 23:23:44,876 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. 2018-06-21 23:23:44,876 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@30f0c6fa 2018-06-21 23:23:44,876 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. 2018-06-21 23:23:44,876 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@30f0c6fa 2018-06-21 23:23:44,876 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. 2018-06-21 23:23:44,876 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@30f0c6fa 2018-06-21 23:23:44,876 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(121): Starting fs shutdown hook thread. 2018-06-21 23:23:44,877 INFO [Thread-3] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished.