2018-12-12 22:06:33,128 DEBUG [main] hbase.HBaseTestingUtility(351): Setting hbase.rootdir to /Users/stack/checkouts/hbase.git/hbase-server/target/test-data/84713365-f84d-f09a-5162-45c6ce274080 2018-12-12 22:06:33,135 INFO [Time-limited test] hbase.HBaseTestingUtility(961): Starting up minicluster with 1 master(s) and 2 regionserver(s) and 2 datanode(s) 2018-12-12 22:06:33,135 INFO [Time-limited test] hbase.HBaseZKTestingUtility(85): Created new mini-cluster data directory: /Users/stack/checkouts/hbase.git/hbase-server/target/test-data/84713365-f84d-f09a-5162-45c6ce274080/cluster_313093fd-5b34-aebf-822e-b59b19ad7940, deleteOnExit=true 2018-12-12 22:06:33,135 INFO [Time-limited test] hbase.HBaseTestingUtility(976): STARTING DFS 2018-12-12 22:06:33,137 INFO [Time-limited test] hbase.HBaseTestingUtility(753): Setting test.cache.data to /Users/stack/checkouts/hbase.git/hbase-server/target/test-data/84713365-f84d-f09a-5162-45c6ce274080/cache_data in system properties and HBase conf 2018-12-12 22:06:33,137 INFO [Time-limited test] hbase.HBaseTestingUtility(753): Setting hadoop.tmp.dir to /Users/stack/checkouts/hbase.git/hbase-server/target/test-data/84713365-f84d-f09a-5162-45c6ce274080/hadoop_tmp in system properties and HBase conf 2018-12-12 22:06:33,138 INFO [Time-limited test] hbase.HBaseTestingUtility(753): Setting hadoop.log.dir to /Users/stack/checkouts/hbase.git/hbase-server/target/test-data/84713365-f84d-f09a-5162-45c6ce274080/hadoop_logs in system properties and HBase conf 2018-12-12 22:06:33,139 INFO [Time-limited test] hbase.HBaseTestingUtility(753): Setting mapreduce.cluster.local.dir to /Users/stack/checkouts/hbase.git/hbase-server/target/test-data/84713365-f84d-f09a-5162-45c6ce274080/mapred_local in system properties and HBase conf 2018-12-12 22:06:33,139 INFO [Time-limited test] hbase.HBaseTestingUtility(753): Setting mapreduce.cluster.temp.dir to /Users/stack/checkouts/hbase.git/hbase-server/target/test-data/84713365-f84d-f09a-5162-45c6ce274080/mapred_temp in system properties and HBase conf 2018-12-12 22:06:33,140 INFO [Time-limited test] hbase.HBaseTestingUtility(744): read short circuit is OFF 2018-12-12 22:06:33,229 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-12-12 22:06:33,461 DEBUG [Time-limited test] fs.HFileSystem(317): The file system is not a DistributedFileSystem. Skipping on block location reordering Formatting using clusterid: testClusterID 2018-12-12 22:06:34,211 WARN [Time-limited test] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2018-12-12 22:06:34,330 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2018-12-12 22:06:34,395 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2018-12-12 22:06:34,412 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/Users/stack/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.7.7/hadoop-hdfs-2.7.7-tests.jar!/webapps/hdfs to /var/folders/rk/jqvccpy53vq7s6mfrfm63ss80000gn/T/Jetty_localhost_54900_hdfs____.64peu4/webapp 2018-12-12 22:06:34,531 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:54900 2018-12-12 22:06:34,999 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2018-12-12 22:06:35,002 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/Users/stack/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.7.7/hadoop-hdfs-2.7.7-tests.jar!/webapps/datanode to /var/folders/rk/jqvccpy53vq7s6mfrfm63ss80000gn/T/Jetty_localhost_54903_datanode____.1ighp8/webapp 2018-12-12 22:06:35,074 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:54903 2018-12-12 22:06:35,285 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2018-12-12 22:06:35,288 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/Users/stack/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.7.7/hadoop-hdfs-2.7.7-tests.jar!/webapps/datanode to /var/folders/rk/jqvccpy53vq7s6mfrfm63ss80000gn/T/Jetty_localhost_54907_datanode____.kx72jc/webapp 2018-12-12 22:06:35,365 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:54907 2018-12-12 22:06:35,632 ERROR [DataNode: [[[DISK]file:/Users/stack/checkouts/hbase.git/hbase-server/target/test-data/84713365-f84d-f09a-5162-45c6ce274080/cluster_313093fd-5b34-aebf-822e-b59b19ad7940/dfs/data/data1/, [DISK]file:/Users/stack/checkouts/hbase.git/hbase-server/target/test-data/84713365-f84d-f09a-5162-45c6ce274080/cluster_313093fd-5b34-aebf-822e-b59b19ad7940/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:54901] datanode.DirectoryScanner(430): dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000 2018-12-12 22:06:35,632 ERROR [DataNode: [[[DISK]file:/Users/stack/checkouts/hbase.git/hbase-server/target/test-data/84713365-f84d-f09a-5162-45c6ce274080/cluster_313093fd-5b34-aebf-822e-b59b19ad7940/dfs/data/data3/, [DISK]file:/Users/stack/checkouts/hbase.git/hbase-server/target/test-data/84713365-f84d-f09a-5162-45c6ce274080/cluster_313093fd-5b34-aebf-822e-b59b19ad7940/dfs/data/data4/]] heartbeating to localhost/127.0.0.1:54901] datanode.DirectoryScanner(430): dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000 2018-12-12 22:06:35,692 INFO [Block report processor] blockmanagement.BlockManager(1930): BLOCK* processReport 0x1a0d1c517d82: from storage DS-572a8465-16cf-4b8a-9cb7-4866d7586234 node DatanodeRegistration(127.0.0.1:54906, datanodeUuid=e5b187b8-acc4-4e11-bf32-fe8e06a333d5, infoPort=54909, infoSecurePort=0, ipcPort=54910, storageInfo=lv=-56;cid=testClusterID;nsid=1341435200;c=0), blocks: 0, hasStaleStorage: true, processing time: 1 msecs 2018-12-12 22:06:35,692 INFO [Block report processor] blockmanagement.BlockManager(1930): BLOCK* processReport 0x1a0d1c517a0a: from storage DS-dfa0bd22-3ff7-4df8-ab3c-2883ad1c4339 node DatanodeRegistration(127.0.0.1:54902, datanodeUuid=a7674cfa-41f1-4a17-84cc-105fc54d1b36, infoPort=54904, infoSecurePort=0, ipcPort=54905, storageInfo=lv=-56;cid=testClusterID;nsid=1341435200;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs 2018-12-12 22:06:35,693 INFO [Block report processor] blockmanagement.BlockManager(1930): BLOCK* processReport 0x1a0d1c517d82: from storage DS-062f61d8-8f41-4a9e-a2ce-6913541fccba node DatanodeRegistration(127.0.0.1:54906, datanodeUuid=e5b187b8-acc4-4e11-bf32-fe8e06a333d5, infoPort=54909, infoSecurePort=0, ipcPort=54910, storageInfo=lv=-56;cid=testClusterID;nsid=1341435200;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs 2018-12-12 22:06:35,693 INFO [Block report processor] blockmanagement.BlockManager(1930): BLOCK* processReport 0x1a0d1c517a0a: from storage DS-b8b11ce4-1484-4801-8b0b-e3a38c78a69a node DatanodeRegistration(127.0.0.1:54902, datanodeUuid=a7674cfa-41f1-4a17-84cc-105fc54d1b36, infoPort=54904, infoSecurePort=0, ipcPort=54905, storageInfo=lv=-56;cid=testClusterID;nsid=1341435200;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs 2018-12-12 22:06:35,795 DEBUG [Time-limited test] hbase.HBaseTestingUtility(679): Setting hbase.rootdir to /Users/stack/checkouts/hbase.git/hbase-server/target/test-data/84713365-f84d-f09a-5162-45c6ce274080 2018-12-12 22:06:35,834 ERROR [Time-limited test] server.ZooKeeperServer(472): ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes 2018-12-12 22:06:35,851 INFO [Time-limited test] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran successful 'stat' on client port=63470 2018-12-12 22:06:35,862 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-12-12 22:06:35,863 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-12-12 22:06:36,107 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54906 is added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b8b11ce4-1484-4801-8b0b-e3a38c78a69a:NORMAL:127.0.0.1:54902|RBW], ReplicaUC[[DISK]DS-062f61d8-8f41-4a9e-a2ce-6913541fccba:NORMAL:127.0.0.1:54906|RBW]]} size 7 2018-12-12 22:06:36,108 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54902 is added to blk_1073741825_1001 size 7 2018-12-12 22:06:36,521 INFO [Time-limited test] util.FSUtils(515): Created version file at hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524 with version=8 2018-12-12 22:06:36,521 INFO [Time-limited test] hbase.HBaseTestingUtility(1242): Setting hbase.fs.tmp.dir to hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/hbase-staging 2018-12-12 22:06:36,709 INFO [Time-limited test] metrics.MetricRegistriesLoader(66): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2018-12-12 22:06:37,097 INFO [Time-limited test] client.ConnectionUtils(122): master/kalashnikov:0 server-side Connection retries=45 2018-12-12 22:06:37,122 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=50, handlerCount=5 2018-12-12 22:06:37,124 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated priority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=50, handlerCount=6 2018-12-12 22:06:37,124 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=50, handlerCount=3 2018-12-12 22:06:37,212 INFO [Time-limited test] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientService, hbase.pb.AdminService 2018-12-12 22:06:37,341 DEBUG [Time-limited test] util.ClassSize(229): Using Unsafe to estimate memory layout 2018-12-12 22:06:37,403 INFO [Time-limited test] ipc.NettyRpcServer(110): Bind to /192.168.1.137:54915 2018-12-12 22:06:37,417 INFO [Time-limited test] hfile.CacheConfig(263): Created cacheConfig: CacheConfig:disabled 2018-12-12 22:06:37,418 INFO [Time-limited test] hfile.CacheConfig(263): Created cacheConfig: CacheConfig:disabled 2018-12-12 22:06:37,421 DEBUG [Time-limited test] mob.MobFileCache(123): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2018-12-12 22:06:37,423 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-12-12 22:06:37,425 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-12-12 22:06:37,625 INFO [Time-limited test] zookeeper.RecoverableZooKeeper(106): Process identifier=master:54915 connecting to ZooKeeper ensemble=localhost:63470 2018-12-12 22:06:37,651 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:549150x0, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-12-12 22:06:37,652 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(543): master:54915-0x167a62c89310000 connected 2018-12-12 22:06:37,676 DEBUG [Time-limited test] zookeeper.ZKUtil(357): master:54915-0x167a62c89310000, quorum=localhost:63470, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-12-12 22:06:37,677 DEBUG [Time-limited test] zookeeper.ZKUtil(357): master:54915-0x167a62c89310000, quorum=localhost:63470, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-12-12 22:06:37,684 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=5 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=54915 2018-12-12 22:06:37,685 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=6 with threadPrefix=priority.FPBQ.Fifo, numCallQueues=1, port=54915 2018-12-12 22:06:37,686 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=54915 2018-12-12 22:06:37,693 INFO [Time-limited test] master.HMaster(504): hbase.rootdir=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524, hbase.cluster.distributed=false 2018-12-12 22:06:37,762 INFO [Time-limited test] client.ConnectionUtils(122): regionserver/kalashnikov:0 server-side Connection retries=45 2018-12-12 22:06:37,762 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=50, handlerCount=5 2018-12-12 22:06:37,762 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated priority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=50, handlerCount=6 2018-12-12 22:06:37,762 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=50, handlerCount=3 2018-12-12 22:06:37,766 INFO [Time-limited test] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-12-12 22:06:37,767 INFO [Time-limited test] io.ByteBufferPool(83): Created with bufferSize=64 KB and maxPoolSize=320 B 2018-12-12 22:06:37,770 INFO [Time-limited test] ipc.NettyRpcServer(110): Bind to /192.168.1.137:54917 2018-12-12 22:06:37,770 INFO [Time-limited test] hfile.CacheConfig(575): Allocating onheap LruBlockCache size=995.60 MB, blockSize=64 KB 2018-12-12 22:06:37,776 INFO [Time-limited test] hfile.CacheConfig(263): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-12-12 22:06:37,777 INFO [Time-limited test] hfile.CacheConfig(263): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-12-12 22:06:37,778 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-12-12 22:06:37,780 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-12-12 22:06:37,782 INFO [Time-limited test] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:54917 connecting to ZooKeeper ensemble=localhost:63470 2018-12-12 22:06:37,786 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:549170x0, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-12-12 22:06:37,787 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(543): regionserver:54917-0x167a62c89310001 connected 2018-12-12 22:06:37,787 DEBUG [Time-limited test] zookeeper.ZKUtil(357): regionserver:54917-0x167a62c89310001, quorum=localhost:63470, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-12-12 22:06:37,788 DEBUG [Time-limited test] zookeeper.ZKUtil(357): regionserver:54917-0x167a62c89310001, quorum=localhost:63470, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-12-12 22:06:37,789 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=5 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=54917 2018-12-12 22:06:37,790 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=6 with threadPrefix=priority.FPBQ.Fifo, numCallQueues=1, port=54917 2018-12-12 22:06:37,791 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=54917 2018-12-12 22:06:37,806 INFO [Time-limited test] client.ConnectionUtils(122): regionserver/kalashnikov:0 server-side Connection retries=45 2018-12-12 22:06:37,806 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=50, handlerCount=5 2018-12-12 22:06:37,806 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated priority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=50, handlerCount=6 2018-12-12 22:06:37,806 INFO [Time-limited test] ipc.RpcExecutor(148): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=50, handlerCount=3 2018-12-12 22:06:37,807 INFO [Time-limited test] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-12-12 22:06:37,807 INFO [Time-limited test] io.ByteBufferPool(83): Created with bufferSize=64 KB and maxPoolSize=320 B 2018-12-12 22:06:37,808 INFO [Time-limited test] ipc.NettyRpcServer(110): Bind to /192.168.1.137:54919 2018-12-12 22:06:37,809 INFO [Time-limited test] hfile.CacheConfig(263): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-12-12 22:06:37,810 INFO [Time-limited test] hfile.CacheConfig(263): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-12-12 22:06:37,811 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-12-12 22:06:37,813 INFO [Time-limited test] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-12-12 22:06:37,815 INFO [Time-limited test] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:54919 connecting to ZooKeeper ensemble=localhost:63470 2018-12-12 22:06:37,818 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:549190x0, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-12-12 22:06:37,818 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(543): regionserver:54919-0x167a62c89310002 connected 2018-12-12 22:06:37,819 DEBUG [Time-limited test] zookeeper.ZKUtil(357): regionserver:54919-0x167a62c89310002, quorum=localhost:63470, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-12-12 22:06:37,819 DEBUG [Time-limited test] zookeeper.ZKUtil(357): regionserver:54919-0x167a62c89310002, quorum=localhost:63470, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-12-12 22:06:37,820 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=5 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=54919 2018-12-12 22:06:37,821 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=6 with threadPrefix=priority.FPBQ.Fifo, numCallQueues=1, port=54919 2018-12-12 22:06:37,821 DEBUG [Time-limited test] ipc.RpcExecutor(263): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=54919 2018-12-12 22:06:37,824 INFO [master/kalashnikov:0:becomeActiveMaster] master.HMaster(2244): Adding backup master ZNode /hbase/backup-masters/kalashnikov,54915,1544681196595 2018-12-12 22:06:37,832 DEBUG [master/kalashnikov:0:becomeActiveMaster] zookeeper.ZKUtil(355): master:54915-0x167a62c89310000, quorum=localhost:63470, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/kalashnikov,54915,1544681196595 2018-12-12 22:06:37,852 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:54915-0x167a62c89310000, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-12-12 22:06:37,852 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:54919-0x167a62c89310002, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-12-12 22:06:37,852 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:54917-0x167a62c89310001, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-12-12 22:06:37,854 DEBUG [master/kalashnikov:0:becomeActiveMaster] zookeeper.ZKUtil(355): master:54915-0x167a62c89310000, quorum=localhost:63470, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-12-12 22:06:37,854 INFO [master/kalashnikov:0:becomeActiveMaster] master.ActiveMasterManager(172): Deleting ZNode for /hbase/backup-masters/kalashnikov,54915,1544681196595 from backup master directory 2018-12-12 22:06:37,858 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): master:54915-0x167a62c89310000, quorum=localhost:63470, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-12-12 22:06:37,858 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:54915-0x167a62c89310000, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/kalashnikov,54915,1544681196595 2018-12-12 22:06:37,859 WARN [master/kalashnikov:0:becomeActiveMaster] hbase.ZNodeClearer(63): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-12-12 22:06:37,859 INFO [master/kalashnikov:0:becomeActiveMaster] master.ActiveMasterManager(181): Registered as active master=kalashnikov,54915,1544681196595 2018-12-12 22:06:37,929 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54906 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dfa0bd22-3ff7-4df8-ab3c-2883ad1c4339:NORMAL:127.0.0.1:54902|RBW], ReplicaUC[[DISK]DS-572a8465-16cf-4b8a-9cb7-4866d7586234:NORMAL:127.0.0.1:54906|RBW]]} size 0 2018-12-12 22:06:37,930 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54902 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dfa0bd22-3ff7-4df8-ab3c-2883ad1c4339:NORMAL:127.0.0.1:54902|RBW], ReplicaUC[[DISK]DS-572a8465-16cf-4b8a-9cb7-4866d7586234:NORMAL:127.0.0.1:54906|RBW]]} size 0 2018-12-12 22:06:37,932 DEBUG [master/kalashnikov:0:becomeActiveMaster] util.FSUtils(667): Created cluster ID file at hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/hbase.id with ID: 7f0e452f-b57d-4360-bdcb-ba2bc88e5eab 2018-12-12 22:06:37,962 INFO [master/kalashnikov:0:becomeActiveMaster] master.MasterFileSystem(396): BOOTSTRAP: creating hbase:meta region 2018-12-12 22:06:37,967 INFO [master/kalashnikov:0:becomeActiveMaster] regionserver.HRegion(7003): creating HRegion hbase:meta HTD == 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, {NAME => 'info', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'false', BLOCKSIZE => '8192'}, {NAME => 'rep_barrier', VERSIONS => '2147483647', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'table', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524 Table name == hbase:meta 2018-12-12 22:06:37,994 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54902 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-062f61d8-8f41-4a9e-a2ce-6913541fccba:NORMAL:127.0.0.1:54906|RBW], ReplicaUC[[DISK]DS-b8b11ce4-1484-4801-8b0b-e3a38c78a69a:NORMAL:127.0.0.1:54902|RBW]]} size 0 2018-12-12 22:06:37,996 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54906 is added to blk_1073741827_1003 size 32 2018-12-12 22:06:38,001 DEBUG [master/kalashnikov:0:becomeActiveMaster] regionserver.HRegion(833): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-12-12 22:06:38,029 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/info 2018-12-12 22:06:38,049 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(237): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-12-12 22:06:38,059 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-12-12 22:06:38,077 INFO [StoreOpener-1588230740-1] regionserver.HStore(332): Store=info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2018-12-12 22:06:38,080 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/rep_barrier 2018-12-12 22:06:38,082 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(237): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-12-12 22:06:38,083 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-12-12 22:06:38,084 INFO [StoreOpener-1588230740-1] regionserver.HStore(332): Store=rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2018-12-12 22:06:38,087 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/table 2018-12-12 22:06:38,088 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(237): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-12-12 22:06:38,088 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-12-12 22:06:38,089 INFO [StoreOpener-1588230740-1] regionserver.HStore(332): Store=table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2018-12-12 22:06:38,098 DEBUG [master/kalashnikov:0:becomeActiveMaster] regionserver.HRegion(4571): Found 0 recovered edits file(s) under hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740 2018-12-12 22:06:38,099 DEBUG [master/kalashnikov:0:becomeActiveMaster] regionserver.HRegion(4571): Found 0 recovered edits file(s) under hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/hbase/meta/1588230740 2018-12-12 22:06:38,110 DEBUG [master/kalashnikov:0:becomeActiveMaster] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7M)) instead. 2018-12-12 22:06:38,111 DEBUG [master/kalashnikov:0:becomeActiveMaster] regionserver.HRegion(998): writing seq id for 1588230740 2018-12-12 22:06:38,116 DEBUG [master/kalashnikov:0:becomeActiveMaster] wal.WALSplitter(695): Wrote file=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-12-12 22:06:38,116 INFO [master/kalashnikov:0:becomeActiveMaster] regionserver.HRegion(1002): Opened 1588230740; next sequenceid=2 2018-12-12 22:06:38,116 DEBUG [master/kalashnikov:0:becomeActiveMaster] regionserver.HRegion(1541): Closing 1588230740, disabling compactions & flushes 2018-12-12 22:06:38,116 DEBUG [master/kalashnikov:0:becomeActiveMaster] regionserver.HRegion(1581): Updates disabled for region hbase:meta,,1.1588230740 2018-12-12 22:06:38,117 INFO [master/kalashnikov:0:becomeActiveMaster] regionserver.HRegion(1698): Closed hbase:meta,,1.1588230740 2018-12-12 22:06:38,146 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54902 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-062f61d8-8f41-4a9e-a2ce-6913541fccba:NORMAL:127.0.0.1:54906|RBW], ReplicaUC[[DISK]DS-dfa0bd22-3ff7-4df8-ab3c-2883ad1c4339:NORMAL:127.0.0.1:54902|FINALIZED]]} size 0 2018-12-12 22:06:38,147 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54906 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dfa0bd22-3ff7-4df8-ab3c-2883ad1c4339:NORMAL:127.0.0.1:54902|FINALIZED], ReplicaUC[[DISK]DS-572a8465-16cf-4b8a-9cb7-4866d7586234:NORMAL:127.0.0.1:54906|FINALIZED]]} size 0 2018-12-12 22:06:38,151 DEBUG [master/kalashnikov:0:becomeActiveMaster] util.FSTableDescriptors(684): Wrote into hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2018-12-12 22:06:38,174 INFO [master/kalashnikov:0:becomeActiveMaster] fs.HFileSystem(348): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-12-12 22:06:38,185 INFO [master/kalashnikov:0:becomeActiveMaster] coordination.ZKSplitLogManagerCoordination(494): Found 0 orphan tasks and 0 rescan nodes 2018-12-12 22:06:38,241 INFO [master/kalashnikov:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x6d7efced to localhost:63470 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-12-12 22:06:38,267 DEBUG [master/kalashnikov:0:becomeActiveMaster] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10cf0467, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-12-12 22:06:38,304 INFO [master/kalashnikov:0:becomeActiveMaster] procedure2.ProcedureExecutor(588): Starting 16 core workers (bigger of cpus/4 or 16) with max (burst) worker count=160, start 1 urgent thread(s) 2018-12-12 22:06:38,306 DEBUG [master/kalashnikov:0:becomeActiveMaster] wal.WALProcedureStore(402): Starting WAL Procedure Store lease recovery 2018-12-12 22:06:38,312 DEBUG [master/kalashnikov:0:becomeActiveMaster] util.CommonFSUtils$DfsBuilderUtility(877): org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder not available, will not use builder API for file creation. 2018-12-12 22:06:38,314 WARN [master/kalashnikov:0:becomeActiveMaster] util.CommonFSUtils$StreamCapabilities(994): Your Hadoop installation does not include the StreamCapabilities class from HDFS-11644, so we will skip checking if any FSDataOutputStreams actually support hflush/hsync. If you are running on top of HDFS this probably just means you have an older version and this can be ignored. If you are running on top of an alternate FileSystem implementation you should manually verify that hflush and hsync are implemented; otherwise you risk data loss and hard to diagnose errors when our assumptions are violated. 2018-12-12 22:06:38,315 DEBUG [master/kalashnikov:0:becomeActiveMaster] util.CommonFSUtils$StreamCapabilities(1001): The first request to check for StreamCapabilities came from this stacktrace. java.lang.ClassNotFoundException: org.apache.hadoop.fs.StreamCapabilities at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.util.CommonFSUtils$StreamCapabilities.(CommonFSUtils.java:990) at org.apache.hadoop.hbase.util.CommonFSUtils.hasCapability(CommonFSUtils.java:1028) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.rollWriter(WALProcedureStore.java:1085) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:423) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.init(ProcedureExecutor.java:611) at org.apache.hadoop.hbase.master.HMaster.createProcedureExecutor(HMaster.java:1457) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:895) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2264) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:581) at java.lang.Thread.run(Thread.java:748) 2018-12-12 22:06:38,319 INFO [master/kalashnikov:0:becomeActiveMaster] wal.WALProcedureStore(1130): Rolled new Procedure Store WAL, id=1 2018-12-12 22:06:38,320 DEBUG [master/kalashnikov:0:becomeActiveMaster] wal.WALProcedureStore(437): Lease acquired for flushLogId=1 2018-12-12 22:06:38,320 INFO [master/kalashnikov:0:becomeActiveMaster] procedure2.ProcedureExecutor(613): Recovered WALProcedureStore lease in 13msec 2018-12-12 22:06:38,321 DEBUG [master/kalashnikov:0:becomeActiveMaster] wal.WALProcedureStore(455): No state logs to replay. 2018-12-12 22:06:38,321 INFO [master/kalashnikov:0:becomeActiveMaster] procedure2.ProcedureExecutor(627): Loaded WALProcedureStore in 0msec 2018-12-12 22:06:38,321 INFO [master/kalashnikov:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(97): Instantiated, coreThreads=128 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2018-12-12 22:06:38,346 DEBUG [master/kalashnikov:0:becomeActiveMaster] zookeeper.ZKUtil(614): master:54915-0x167a62c89310000, quorum=localhost:63470, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-12-12 22:06:38,357 INFO [master/kalashnikov:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2018-12-12 22:06:38,381 INFO [master/kalashnikov:0:becomeActiveMaster] balancer.BaseLoadBalancer(1035): slop=0.001, systemTablesOnMaster=false 2018-12-12 22:06:38,401 INFO [master/kalashnikov:0:becomeActiveMaster] balancer.StochasticLoadBalancer(213): Loaded config; maxSteps=1000000, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, etc. 2018-12-12 22:06:38,405 DEBUG [master/kalashnikov:0:becomeActiveMaster] zookeeper.ZKUtil(357): master:54915-0x167a62c89310000, quorum=localhost:63470, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2018-12-12 22:06:38,406 DEBUG [master/kalashnikov:0:becomeActiveMaster] zookeeper.ZKUtil(357): master:54915-0x167a62c89310000, quorum=localhost:63470, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2018-12-12 22:06:38,410 DEBUG [master/kalashnikov:0:becomeActiveMaster] zookeeper.ZKUtil(357): master:54915-0x167a62c89310000, quorum=localhost:63470, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2018-12-12 22:06:38,410 DEBUG [master/kalashnikov:0:becomeActiveMaster] zookeeper.ZKUtil(357): master:54915-0x167a62c89310000, quorum=localhost:63470, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2018-12-12 22:06:38,426 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:54915-0x167a62c89310000, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-12-12 22:06:38,426 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:54917-0x167a62c89310001, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-12-12 22:06:38,426 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:54919-0x167a62c89310002, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-12-12 22:06:38,426 INFO [master/kalashnikov:0:becomeActiveMaster] master.HMaster(795): Active/primary master=kalashnikov,54915,1544681196595, sessionid=0x167a62c89310000, setting cluster-up flag (Was=false) 2018-12-12 22:06:38,443 DEBUG [master/kalashnikov:0:becomeActiveMaster] procedure.ZKProcedureUtil(272): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2018-12-12 22:06:38,445 DEBUG [master/kalashnikov:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(250): Starting controller for procedure member=kalashnikov,54915,1544681196595 2018-12-12 22:06:38,452 DEBUG [master/kalashnikov:0:becomeActiveMaster] procedure.ZKProcedureUtil(272): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2018-12-12 22:06:38,453 DEBUG [master/kalashnikov:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(250): Starting controller for procedure member=kalashnikov,54915,1544681196595 2018-12-12 22:06:38,537 INFO [RS:1;kalashnikov:54919] regionserver.HRegionServer(878): ClusterId : 7f0e452f-b57d-4360-bdcb-ba2bc88e5eab 2018-12-12 22:06:38,537 INFO [RS:0;kalashnikov:54917] regionserver.HRegionServer(878): ClusterId : 7f0e452f-b57d-4360-bdcb-ba2bc88e5eab 2018-12-12 22:06:38,541 DEBUG [RS:0;kalashnikov:54917] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initializing 2018-12-12 22:06:38,541 DEBUG [RS:1;kalashnikov:54919] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initializing 2018-12-12 22:06:38,546 DEBUG [RS:0;kalashnikov:54917] procedure.RegionServerProcedureManagerHost(47): Procedure flush-table-proc initialized 2018-12-12 22:06:38,546 DEBUG [RS:0;kalashnikov:54917] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initializing 2018-12-12 22:06:38,546 DEBUG [RS:1;kalashnikov:54919] procedure.RegionServerProcedureManagerHost(47): Procedure flush-table-proc initialized 2018-12-12 22:06:38,546 DEBUG [RS:1;kalashnikov:54919] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initializing 2018-12-12 22:06:38,549 DEBUG [RS:0;kalashnikov:54917] procedure.RegionServerProcedureManagerHost(47): Procedure online-snapshot initialized 2018-12-12 22:06:38,549 DEBUG [RS:1;kalashnikov:54919] procedure.RegionServerProcedureManagerHost(47): Procedure online-snapshot initialized 2018-12-12 22:06:38,550 INFO [RS:0;kalashnikov:54917] zookeeper.ReadOnlyZKClient(139): Connect 0x2f9816cf to localhost:63470 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-12-12 22:06:38,550 INFO [RS:1;kalashnikov:54919] zookeeper.ReadOnlyZKClient(139): Connect 0x45ed407e to localhost:63470 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-12-12 22:06:38,556 DEBUG [RS:1;kalashnikov:54919] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4b2aa6b3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-12-12 22:06:38,556 DEBUG [RS:0;kalashnikov:54917] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@215fed4c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-12-12 22:06:38,557 DEBUG [RS:1;kalashnikov:54919] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4f152b0e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=kalashnikov/192.168.1.137:0 2018-12-12 22:06:38,557 DEBUG [RS:0;kalashnikov:54917] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@25607dbf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=kalashnikov/192.168.1.137:0 2018-12-12 22:06:38,560 DEBUG [RS:0;kalashnikov:54917] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:0;kalashnikov:54917 2018-12-12 22:06:38,560 DEBUG [RS:1;kalashnikov:54919] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:1;kalashnikov:54919 2018-12-12 22:06:38,570 INFO [master/kalashnikov:0:becomeActiveMaster] master.HMaster(946): hbase:meta {1588230740 state=OFFLINE, ts=1544681198350, server=null} 2018-12-12 22:06:38,570 INFO [RS:1;kalashnikov:54919] regionserver.RegionServerCoprocessorHost(67): System coprocessor loading is enabled 2018-12-12 22:06:38,570 INFO [RS:1;kalashnikov:54919] regionserver.RegionServerCoprocessorHost(68): Table coprocessor loading is enabled 2018-12-12 22:06:38,570 INFO [RS:0;kalashnikov:54917] regionserver.RegionServerCoprocessorHost(67): System coprocessor loading is enabled 2018-12-12 22:06:38,570 INFO [RS:0;kalashnikov:54917] regionserver.RegionServerCoprocessorHost(68): Table coprocessor loading is enabled 2018-12-12 22:06:38,570 DEBUG [RS:0;kalashnikov:54917] regionserver.HRegionServer(947): About to register with Master. 2018-12-12 22:06:38,570 DEBUG [RS:1;kalashnikov:54919] regionserver.HRegionServer(947): About to register with Master. 2018-12-12 22:06:38,573 INFO [RS:1;kalashnikov:54919] regionserver.HRegionServer(2593): reportForDuty to master=kalashnikov,54915,1544681196595 with port=54919, startcode=1544681197805 2018-12-12 22:06:38,573 INFO [RS:0;kalashnikov:54917] regionserver.HRegionServer(2593): reportForDuty to master=kalashnikov,54915,1544681196595 with port=54917, startcode=1544681197760 2018-12-12 22:06:38,647 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(556): Connection from 192.168.1.137:54932, version=2.1.2, sasl=false, ugi=stack.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2018-12-12 22:06:38,647 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(556): Connection from 192.168.1.137:54931, version=2.1.2, sasl=false, ugi=stack.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2018-12-12 22:06:38,686 DEBUG [RS:0;kalashnikov:54917] regionserver.HRegionServer(2613): Master is not running yet 2018-12-12 22:06:38,686 DEBUG [RS:1;kalashnikov:54919] regionserver.HRegionServer(2613): Master is not running yet 2018-12-12 22:06:38,686 WARN [RS:0;kalashnikov:54917] regionserver.HRegionServer(955): reportForDuty failed; sleeping 1000 ms and then retrying. 2018-12-12 22:06:38,686 WARN [RS:1;kalashnikov:54919] regionserver.HRegionServer(955): reportForDuty failed; sleeping 1000 ms and then retrying. 2018-12-12 22:06:38,721 DEBUG [master/kalashnikov:0:becomeActiveMaster] procedure2.ProcedureExecutor(1092): Stored pid=1, state=RUNNABLE:INIT_META_ASSIGN_META; InitMetaProcedure table=hbase:meta 2018-12-12 22:06:38,723 DEBUG [master/kalashnikov:0:becomeActiveMaster] procedure.MasterProcedureScheduler(356): Add MetaQueue(hbase:meta, xlock=false sharedLock=0 size=1) to run queue because: the exclusive lock is not held by anyone when adding pid=1, state=RUNNABLE:INIT_META_ASSIGN_META; InitMetaProcedure table=hbase:meta 2018-12-12 22:06:38,732 DEBUG [master/kalashnikov:0:becomeActiveMaster] executor.ExecutorService(92): Starting executor service name=MASTER_OPEN_REGION-master/kalashnikov:0, corePoolSize=5, maxPoolSize=5 2018-12-12 22:06:38,732 DEBUG [master/kalashnikov:0:becomeActiveMaster] executor.ExecutorService(92): Starting executor service name=MASTER_CLOSE_REGION-master/kalashnikov:0, corePoolSize=5, maxPoolSize=5 2018-12-12 22:06:38,732 DEBUG [master/kalashnikov:0:becomeActiveMaster] executor.ExecutorService(92): Starting executor service name=MASTER_SERVER_OPERATIONS-master/kalashnikov:0, corePoolSize=5, maxPoolSize=5 2018-12-12 22:06:38,732 DEBUG [master/kalashnikov:0:becomeActiveMaster] executor.ExecutorService(92): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/kalashnikov:0, corePoolSize=5, maxPoolSize=5 2018-12-12 22:06:38,733 DEBUG [master/kalashnikov:0:becomeActiveMaster] executor.ExecutorService(92): Starting executor service name=M_LOG_REPLAY_OPS-master/kalashnikov:0, corePoolSize=10, maxPoolSize=10 2018-12-12 22:06:38,733 DEBUG [master/kalashnikov:0:becomeActiveMaster] executor.ExecutorService(92): Starting executor service name=MASTER_TABLE_OPERATIONS-master/kalashnikov:0, corePoolSize=1, maxPoolSize=1 2018-12-12 22:06:38,733 DEBUG [master/kalashnikov:0:becomeActiveMaster] procedure2.ProcedureExecutor(640): Start workers 16, urgent workers 1 2018-12-12 22:06:38,734 DEBUG [PEWorker-1] procedure.MasterProcedureScheduler(366): Remove MetaQueue(hbase:meta, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=1, state=RUNNABLE:INIT_META_ASSIGN_META; InitMetaProcedure table=hbase:meta 2018-12-12 22:06:38,736 INFO [master/kalashnikov:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1544681228735 2018-12-12 22:06:38,737 DEBUG [PEWorker-1] procedure.MasterProcedureScheduler(366): Remove TableQueue(hbase:meta, xlock=true (1) sharedLock=0 size=0) from run queue because: pid=1, state=RUNNABLE:INIT_META_ASSIGN_META; InitMetaProcedure table=hbase:meta held the exclusive lock 2018-12-12 22:06:38,739 INFO [master/kalashnikov:0:becomeActiveMaster] cleaner.CleanerChore$DirScanPool(90): Cleaner pool size is 2 2018-12-12 22:06:38,740 DEBUG [master/kalashnikov:0:becomeActiveMaster] cleaner.CleanerChore(251): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2018-12-12 22:06:38,740 INFO [master/kalashnikov:0:becomeActiveMaster] zookeeper.RecoverableZooKeeper(106): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:63470 2018-12-12 22:06:38,741 DEBUG [master/kalashnikov:0:becomeActiveMaster] cleaner.CleanerChore(251): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2018-12-12 22:06:38,742 DEBUG [master/kalashnikov:0:becomeActiveMaster] cleaner.CleanerChore(251): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2018-12-12 22:06:38,742 INFO [master/kalashnikov:0:becomeActiveMaster] cleaner.LogCleaner(155): Creating OldWALs cleaners with size=2 2018-12-12 22:06:38,744 DEBUG [master/kalashnikov:0:becomeActiveMaster-EventThread] zookeeper.ZKWatcher(478): replicationLogCleaner0x0, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-12-12 22:06:38,745 DEBUG [master/kalashnikov:0:becomeActiveMaster-EventThread] zookeeper.ZKWatcher(543): replicationLogCleaner-0x167a62c89310006 connected 2018-12-12 22:06:38,747 INFO [PEWorker-1] procedure2.ProcedureExecutor(1758): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740}] 2018-12-12 22:06:38,747 DEBUG [PEWorker-1] procedure2.RootProcedureState(153): Add procedure pid=1, state=WAITING, locked=true; InitMetaProcedure table=hbase:meta as the 0th rollback step 2018-12-12 22:06:38,747 DEBUG [master/kalashnikov:0:becomeActiveMaster] cleaner.CleanerChore(251): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2018-12-12 22:06:38,750 DEBUG [master/kalashnikov:0:becomeActiveMaster] cleaner.CleanerChore(251): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2018-12-12 22:06:38,750 DEBUG [master/kalashnikov:0:becomeActiveMaster] cleaner.CleanerChore(251): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2018-12-12 22:06:38,752 DEBUG [master/kalashnikov:0:becomeActiveMaster] cleaner.HFileCleaner(225): Starting for large file=Thread[master/kalashnikov:0:becomeActiveMaster-HFileCleaner.large.0-1544681198752,5,FailOnTimeoutGroup] 2018-12-12 22:06:38,753 DEBUG [master/kalashnikov:0:becomeActiveMaster] cleaner.HFileCleaner(240): Starting for small files=Thread[master/kalashnikov:0:becomeActiveMaster-HFileCleaner.small.0-1544681198753,5,FailOnTimeoutGroup] 2018-12-12 22:06:38,756 DEBUG [PEWorker-1] procedure.MasterProcedureScheduler(356): Add MetaQueue(hbase:meta, xlock=false sharedLock=0 size=1) to run queue because: the exclusive lock is not held by anyone when adding pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740 2018-12-12 22:06:38,756 DEBUG [PEWorker-2] procedure.MasterProcedureScheduler(366): Remove MetaQueue(hbase:meta, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740 2018-12-12 22:06:38,757 INFO [PEWorker-2] procedure.MasterProcedureScheduler(741): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740 2018-12-12 22:06:38,760 DEBUG [PEWorker-1] procedure.MasterProcedureScheduler(356): Add TableQueue(hbase:meta, xlock=false sharedLock=1 size=0) to run queue because: pid=1, state=WAITING; InitMetaProcedure table=hbase:meta released the exclusive lock 2018-12-12 22:06:38,768 INFO [PEWorker-2] assignment.AssignProcedure(254): Starting pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2018-12-12 22:06:38,768 DEBUG [PEWorker-2] procedure2.RootProcedureState(153): Add procedure pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:meta, region=1588230740 as the 1th rollback step 2018-12-12 22:06:38,922 WARN [master/kalashnikov:0] assignment.AssignmentManager(1686): No servers available; cannot place 1 unassigned regions. 2018-12-12 22:06:39,689 INFO [RS:1;kalashnikov:54919] regionserver.HRegionServer(2593): reportForDuty to master=kalashnikov,54915,1544681196595 with port=54919, startcode=1544681197805 2018-12-12 22:06:39,689 INFO [RS:0;kalashnikov:54917] regionserver.HRegionServer(2593): reportForDuty to master=kalashnikov,54915,1544681196595 with port=54917, startcode=1544681197760 2018-12-12 22:06:39,705 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=54915] master.ServerManager(403): Registering regionserver=kalashnikov,54917,1544681197760 2018-12-12 22:06:39,705 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54915] master.ServerManager(403): Registering regionserver=kalashnikov,54919,1544681197805 2018-12-12 22:06:39,714 DEBUG [RS:0;kalashnikov:54917] regionserver.HRegionServer(1490): Config from master: hbase.rootdir=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524 2018-12-12 22:06:39,714 DEBUG [RS:1;kalashnikov:54919] regionserver.HRegionServer(1490): Config from master: hbase.rootdir=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524 2018-12-12 22:06:39,714 DEBUG [RS:0;kalashnikov:54917] regionserver.HRegionServer(1490): Config from master: fs.defaultFS=hdfs://localhost:54901 2018-12-12 22:06:39,715 DEBUG [RS:0;kalashnikov:54917] regionserver.HRegionServer(1490): Config from master: hbase.master.info.port=-1 2018-12-12 22:06:39,715 DEBUG [RS:1;kalashnikov:54919] regionserver.HRegionServer(1490): Config from master: fs.defaultFS=hdfs://localhost:54901 2018-12-12 22:06:39,715 DEBUG [RS:1;kalashnikov:54919] regionserver.HRegionServer(1490): Config from master: hbase.master.info.port=-1 2018-12-12 22:06:39,721 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:54915-0x167a62c89310000, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-12-12 22:06:39,722 DEBUG [RS:0;kalashnikov:54917] zookeeper.ZKUtil(355): regionserver:54917-0x167a62c89310001, quorum=localhost:63470, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/kalashnikov,54917,1544681197760 2018-12-12 22:06:39,722 DEBUG [RS:1;kalashnikov:54919] zookeeper.ZKUtil(355): regionserver:54919-0x167a62c89310002, quorum=localhost:63470, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/kalashnikov,54919,1544681197805 2018-12-12 22:06:39,722 WARN [RS:0;kalashnikov:54917] hbase.ZNodeClearer(63): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-12-12 22:06:39,722 WARN [RS:1;kalashnikov:54919] hbase.ZNodeClearer(63): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-12-12 22:06:39,724 INFO [RegionServerTracker-0] master.RegionServerTracker(182): RegionServer ephemeral node created, adding [kalashnikov,54919,1544681197805] 2018-12-12 22:06:39,725 INFO [RegionServerTracker-0] master.RegionServerTracker(182): RegionServer ephemeral node created, adding [kalashnikov,54917,1544681197760] 2018-12-12 22:06:39,748 DEBUG [RS:0;kalashnikov:54917] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(421): org.apache.hadoop.hdfs.protocolPB.PBHelperClient not found (Hadoop is pre-2.8.0?); using class org.apache.hadoop.hdfs.protocolPB.PBHelper instead. 2018-12-12 22:06:39,782 DEBUG [RS:0;kalashnikov:54917] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(494): No DfsClientConf class found, should be hadoop 2.7- java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.client.impl.DfsClientConf at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createChecksumCreater(FanOutOneBlockAsyncDFSOutputHelper.java:492) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:556) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:136) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:136) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:198) at org.apache.hadoop.hbase.regionserver.HRegionServer.setupWALAndReplication(HRegionServer.java:1794) at org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1512) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.handleReportForDutyResponse(MiniHBaseCluster.java:156) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:958) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:183) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:129) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:164) at java.lang.Thread.run(Thread.java:748) 2018-12-12 22:06:39,784 DEBUG [RS:0;kalashnikov:54917] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(528): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2018-12-12 22:06:39,790 DEBUG [RS:0;kalashnikov:54917] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(196): No PBHelperClient class found, should be hadoop 2.7- java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.protocolPB.PBHelperClient at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createPBHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:194) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:299) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:137) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:136) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:198) at org.apache.hadoop.hbase.regionserver.HRegionServer.setupWALAndReplication(HRegionServer.java:1794) at org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1512) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.handleReportForDutyResponse(MiniHBaseCluster.java:156) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:958) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:183) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:129) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:164) at java.lang.Thread.run(Thread.java:748) 2018-12-12 22:06:39,792 INFO [RS:0;kalashnikov:54917] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-12-12 22:06:39,792 INFO [RS:1;kalashnikov:54919] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-12-12 22:06:39,795 DEBUG [RS:0;kalashnikov:54917] regionserver.HRegionServer(1801): logDir=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/WALs/kalashnikov,54917,1544681197760 2018-12-12 22:06:39,795 DEBUG [RS:1;kalashnikov:54919] regionserver.HRegionServer(1801): logDir=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/WALs/kalashnikov,54919,1544681197805 2018-12-12 22:06:39,810 DEBUG [RS:1;kalashnikov:54919] zookeeper.ZKUtil(355): regionserver:54919-0x167a62c89310002, quorum=localhost:63470, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/kalashnikov,54919,1544681197805 2018-12-12 22:06:39,810 DEBUG [RS:0;kalashnikov:54917] zookeeper.ZKUtil(355): regionserver:54917-0x167a62c89310001, quorum=localhost:63470, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/kalashnikov,54919,1544681197805 2018-12-12 22:06:39,811 DEBUG [RS:1;kalashnikov:54919] zookeeper.ZKUtil(355): regionserver:54919-0x167a62c89310002, quorum=localhost:63470, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/kalashnikov,54917,1544681197760 2018-12-12 22:06:39,811 DEBUG [RS:0;kalashnikov:54917] zookeeper.ZKUtil(355): regionserver:54917-0x167a62c89310001, quorum=localhost:63470, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/kalashnikov,54917,1544681197760 2018-12-12 22:06:39,815 DEBUG [RS:0;kalashnikov:54917] regionserver.Replication(131): Replication stats-in-log period=300 seconds 2018-12-12 22:06:39,815 DEBUG [RS:1;kalashnikov:54919] regionserver.Replication(131): Replication stats-in-log period=300 seconds 2018-12-12 22:06:39,821 INFO [RS:0;kalashnikov:54917] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-12-12 22:06:39,821 INFO [RS:1;kalashnikov:54919] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-12-12 22:06:39,845 INFO [RS:0;kalashnikov:54917] regionserver.MemStoreFlusher(132): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-12-12 22:06:39,845 INFO [RS:1;kalashnikov:54919] regionserver.MemStoreFlusher(132): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, Offheap=false 2018-12-12 22:06:39,852 INFO [RS:1;kalashnikov:54919] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/second, lower bound 10.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2018-12-12 22:06:39,852 INFO [RS:0;kalashnikov:54917] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/second, lower bound 10.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2018-12-12 22:06:39,854 INFO [RS:1;kalashnikov:54919] regionserver.HRegionServer$CompactionChecker(1690): CompactionChecker runs every PT1S 2018-12-12 22:06:39,854 INFO [RS:0;kalashnikov:54917] regionserver.HRegionServer$CompactionChecker(1690): CompactionChecker runs every PT1S 2018-12-12 22:06:39,863 DEBUG [RS:0;kalashnikov:54917] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-regionserver/kalashnikov:0, corePoolSize=3, maxPoolSize=3 2018-12-12 22:06:39,863 DEBUG [RS:1;kalashnikov:54919] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-regionserver/kalashnikov:0, corePoolSize=3, maxPoolSize=3 2018-12-12 22:06:39,864 DEBUG [RS:0;kalashnikov:54917] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-regionserver/kalashnikov:0, corePoolSize=1, maxPoolSize=1 2018-12-12 22:06:39,864 DEBUG [RS:1;kalashnikov:54919] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-regionserver/kalashnikov:0, corePoolSize=1, maxPoolSize=1 2018-12-12 22:06:39,864 DEBUG [RS:0;kalashnikov:54917] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/kalashnikov:0, corePoolSize=3, maxPoolSize=3 2018-12-12 22:06:39,864 DEBUG [RS:1;kalashnikov:54919] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/kalashnikov:0, corePoolSize=3, maxPoolSize=3 2018-12-12 22:06:39,864 DEBUG [RS:0;kalashnikov:54917] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-regionserver/kalashnikov:0, corePoolSize=3, maxPoolSize=3 2018-12-12 22:06:39,864 DEBUG [RS:1;kalashnikov:54919] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-regionserver/kalashnikov:0, corePoolSize=3, maxPoolSize=3 2018-12-12 22:06:39,864 DEBUG [RS:0;kalashnikov:54917] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-regionserver/kalashnikov:0, corePoolSize=1, maxPoolSize=1 2018-12-12 22:06:39,864 DEBUG [RS:1;kalashnikov:54919] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-regionserver/kalashnikov:0, corePoolSize=1, maxPoolSize=1 2018-12-12 22:06:39,864 DEBUG [RS:0;kalashnikov:54917] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/kalashnikov:0, corePoolSize=2, maxPoolSize=2 2018-12-12 22:06:39,864 DEBUG [RS:1;kalashnikov:54919] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/kalashnikov:0, corePoolSize=2, maxPoolSize=2 2018-12-12 22:06:39,865 DEBUG [RS:0;kalashnikov:54917] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/kalashnikov:0, corePoolSize=10, maxPoolSize=10 2018-12-12 22:06:39,865 DEBUG [RS:1;kalashnikov:54919] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/kalashnikov:0, corePoolSize=10, maxPoolSize=10 2018-12-12 22:06:39,865 DEBUG [RS:0;kalashnikov:54917] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/kalashnikov:0, corePoolSize=3, maxPoolSize=3 2018-12-12 22:06:39,865 DEBUG [RS:1;kalashnikov:54919] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/kalashnikov:0, corePoolSize=3, maxPoolSize=3 2018-12-12 22:06:39,865 DEBUG [RS:0;kalashnikov:54917] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-regionserver/kalashnikov:0, corePoolSize=2, maxPoolSize=2 2018-12-12 22:06:39,865 DEBUG [RS:1;kalashnikov:54919] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-regionserver/kalashnikov:0, corePoolSize=2, maxPoolSize=2 2018-12-12 22:06:39,885 INFO [SplitLogWorker-kalashnikov:54919] regionserver.SplitLogWorker(136): SplitLogWorker kalashnikov,54919,1544681197805 starting 2018-12-12 22:06:39,885 INFO [SplitLogWorker-kalashnikov:54917] regionserver.SplitLogWorker(136): SplitLogWorker kalashnikov,54917,1544681197760 starting 2018-12-12 22:06:39,890 INFO [RS:1;kalashnikov:54919] regionserver.HeapMemoryManager(210): Starting, tuneOn=false 2018-12-12 22:06:39,890 INFO [RS:0;kalashnikov:54917] regionserver.HeapMemoryManager(210): Starting, tuneOn=false 2018-12-12 22:06:39,895 INFO [RS:0;kalashnikov:54917] regionserver.ChunkCreator(499): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 448, initial count 0 2018-12-12 22:06:39,895 INFO [RS:1;kalashnikov:54919] regionserver.ChunkCreator(499): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 448, initial count 0 2018-12-12 22:06:39,897 INFO [RS:1;kalashnikov:54919] regionserver.ChunkCreator(499): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 497, initial count 0 2018-12-12 22:06:39,897 INFO [RS:0;kalashnikov:54917] regionserver.ChunkCreator(499): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 497, initial count 0 2018-12-12 22:06:39,920 INFO [RS:0;kalashnikov:54917] regionserver.HRegionServer(1531): Serving as kalashnikov,54917,1544681197760, RpcServer on kalashnikov/192.168.1.137:54917, sessionid=0x167a62c89310001 2018-12-12 22:06:39,920 INFO [RS:1;kalashnikov:54919] regionserver.HRegionServer(1531): Serving as kalashnikov,54919,1544681197805, RpcServer on kalashnikov/192.168.1.137:54919, sessionid=0x167a62c89310002 2018-12-12 22:06:39,920 DEBUG [RS:1;kalashnikov:54919] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc starting 2018-12-12 22:06:39,920 DEBUG [RS:1;kalashnikov:54919] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager kalashnikov,54919,1544681197805 2018-12-12 22:06:39,920 DEBUG [RS:1;kalashnikov:54919] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'kalashnikov,54919,1544681197805' 2018-12-12 22:06:39,920 DEBUG [RS:1;kalashnikov:54919] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-12-12 22:06:39,920 DEBUG [RS:0;kalashnikov:54917] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc starting 2018-12-12 22:06:39,921 DEBUG [RS:0;kalashnikov:54917] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager kalashnikov,54917,1544681197760 2018-12-12 22:06:39,921 DEBUG [RS:0;kalashnikov:54917] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'kalashnikov,54917,1544681197760' 2018-12-12 22:06:39,922 DEBUG [RS:0;kalashnikov:54917] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-12-12 22:06:39,922 DEBUG [RS:1;kalashnikov:54919] procedure.ZKProcedureMemberRpcs(155): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-12-12 22:06:39,922 DEBUG [RS:0;kalashnikov:54917] procedure.ZKProcedureMemberRpcs(155): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-12-12 22:06:39,922 DEBUG [RS:1;kalashnikov:54919] procedure.RegionServerProcedureManagerHost(55): Procedure flush-table-proc started 2018-12-12 22:06:39,922 DEBUG [RS:0;kalashnikov:54917] procedure.RegionServerProcedureManagerHost(55): Procedure flush-table-proc started 2018-12-12 22:06:39,923 DEBUG [RS:0;kalashnikov:54917] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot starting 2018-12-12 22:06:39,922 DEBUG [RS:1;kalashnikov:54919] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot starting 2018-12-12 22:06:39,923 DEBUG [RS:0;kalashnikov:54917] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager kalashnikov,54917,1544681197760 2018-12-12 22:06:39,923 DEBUG [RS:0;kalashnikov:54917] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'kalashnikov,54917,1544681197760' 2018-12-12 22:06:39,923 DEBUG [RS:0;kalashnikov:54917] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-12-12 22:06:39,923 DEBUG [RS:1;kalashnikov:54919] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager kalashnikov,54919,1544681197805 2018-12-12 22:06:39,923 DEBUG [RS:1;kalashnikov:54919] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'kalashnikov,54919,1544681197805' 2018-12-12 22:06:39,923 DEBUG [RS:1;kalashnikov:54919] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-12-12 22:06:39,924 DEBUG [RS:0;kalashnikov:54917] procedure.ZKProcedureMemberRpcs(155): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-12-12 22:06:39,924 DEBUG [RS:1;kalashnikov:54919] procedure.ZKProcedureMemberRpcs(155): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-12-12 22:06:39,924 DEBUG [RS:0;kalashnikov:54917] procedure.RegionServerProcedureManagerHost(55): Procedure online-snapshot started 2018-12-12 22:06:39,924 INFO [RS:0;kalashnikov:54917] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-12-12 22:06:39,924 INFO [RS:0;kalashnikov:54917] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-12-12 22:06:39,924 DEBUG [RS:1;kalashnikov:54919] procedure.RegionServerProcedureManagerHost(55): Procedure online-snapshot started 2018-12-12 22:06:39,924 INFO [RS:1;kalashnikov:54919] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-12-12 22:06:39,925 INFO [RS:1;kalashnikov:54919] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-12-12 22:06:39,935 DEBUG [master/kalashnikov:0] assignment.AssignmentManager(1707): Processing assignQueue; systemServersCount=2, allServersCount=2 2018-12-12 22:06:39,948 DEBUG [master/kalashnikov:0] procedure.MasterProcedureScheduler(356): Add MetaQueue(hbase:meta, xlock=false sharedLock=0 size=1) to run queue because: pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:meta, region=1588230740 has lock 2018-12-12 22:06:39,948 DEBUG [PEWorker-3] procedure.MasterProcedureScheduler(366): Remove MetaQueue(hbase:meta, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:meta, region=1588230740 2018-12-12 22:06:39,948 WARN [PEWorker-3] assignment.AssignmentManager(1056): Why is ServerStateNode for kalashnikov,54919,1544681197805 empty at this point? Creating... 2018-12-12 22:06:39,952 INFO [PEWorker-3] assignment.AssignProcedure(282): Early suspend! pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=kalashnikov,54919,1544681197805 2018-12-12 22:06:39,952 DEBUG [PEWorker-3] procedure2.RootProcedureState(153): Add procedure pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:meta, region=1588230740 as the 2th rollback step 2018-12-12 22:06:40,964 INFO [RS:1;kalashnikov:54919] wal.AbstractFSWAL(414): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=kalashnikov%2C54919%2C1544681197805, suffix=, logDir=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/WALs/kalashnikov,54919,1544681197805, archiveDir=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/oldWALs 2018-12-12 22:06:40,964 INFO [RS:0;kalashnikov:54917] wal.AbstractFSWAL(414): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=kalashnikov%2C54917%2C1544681197760, suffix=, logDir=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/WALs/kalashnikov,54917,1544681197760, archiveDir=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/oldWALs 2018-12-12 22:06:41,000 DEBUG [RS-EventLoopGroup-4-5] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(783): SASL client skipping handshake in unsecured configuration for addr = /127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:54906,DS-572a8465-16cf-4b8a-9cb7-4866d7586234,DISK] 2018-12-12 22:06:41,002 DEBUG [RS-EventLoopGroup-4-7] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(783): SASL client skipping handshake in unsecured configuration for addr = /127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:54902,DS-b8b11ce4-1484-4801-8b0b-e3a38c78a69a,DISK] 2018-12-12 22:06:41,003 DEBUG [RS-EventLoopGroup-4-4] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(783): SASL client skipping handshake in unsecured configuration for addr = /127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:54906,DS-572a8465-16cf-4b8a-9cb7-4866d7586234,DISK] 2018-12-12 22:06:41,001 DEBUG [RS-EventLoopGroup-4-6] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(783): SASL client skipping handshake in unsecured configuration for addr = /127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:54902,DS-dfa0bd22-3ff7-4df8-ab3c-2883ad1c4339,DISK] 2018-12-12 22:06:41,033 INFO [RS:1;kalashnikov:54919] wal.AbstractFSWAL(672): New WAL /user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/WALs/kalashnikov,54919,1544681197805/kalashnikov%2C54919%2C1544681197805.1544681200975 2018-12-12 22:06:41,033 INFO [RS:0;kalashnikov:54917] wal.AbstractFSWAL(672): New WAL /user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/WALs/kalashnikov,54917,1544681197760/kalashnikov%2C54917%2C1544681197760.1544681200975 2018-12-12 22:06:41,033 DEBUG [RS:1;kalashnikov:54919] wal.AbstractFSWAL(762): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:54906,DS-572a8465-16cf-4b8a-9cb7-4866d7586234,DISK], DatanodeInfoWithStorage[127.0.0.1:54902,DS-dfa0bd22-3ff7-4df8-ab3c-2883ad1c4339,DISK]] 2018-12-12 22:06:41,033 DEBUG [RS:0;kalashnikov:54917] wal.AbstractFSWAL(762): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:54906,DS-572a8465-16cf-4b8a-9cb7-4866d7586234,DISK], DatanodeInfoWithStorage[127.0.0.1:54902,DS-b8b11ce4-1484-4801-8b0b-e3a38c78a69a,DISK]] 2018-12-12 22:06:41,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=54915] procedure.MasterProcedureScheduler(356): Add MetaQueue(hbase:meta, xlock=false sharedLock=0 size=1) to run queue because: pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:meta, region=1588230740 has lock 2018-12-12 22:06:41,053 DEBUG [PEWorker-16] procedure.MasterProcedureScheduler(366): Remove MetaQueue(hbase:meta, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:meta, region=1588230740 2018-12-12 22:06:41,054 INFO [PEWorker-16] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as kalashnikov,54919,1544681197805 2018-12-12 22:06:41,062 DEBUG [PEWorker-16] zookeeper.MetaTableLocator(466): META region location doesn't exist, create it 2018-12-12 22:06:41,063 INFO [PEWorker-16] assignment.RegionTransitionProcedure(267): Dispatch pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:meta, region=1588230740 2018-12-12 22:06:41,063 DEBUG [PEWorker-16] procedure2.RootProcedureState(153): Add procedure pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:meta, region=1588230740 as the 3th rollback step 2018-12-12 22:06:41,231 DEBUG [RSProcedureDispatcher-pool3-t1] master.ServerManager(728): New admin connection to kalashnikov,54919,1544681197805 2018-12-12 22:06:41,243 INFO [RS-EventLoopGroup-4-13] ipc.ServerRpcConnection(556): Connection from 192.168.1.137:54942, version=2.1.2, sasl=false, ugi=stack (auth:SIMPLE), service=AdminService 2018-12-12 22:06:41,261 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=54919] regionserver.RSRpcServices(1987): Open hbase:meta,,1.1588230740 2018-12-12 22:06:41,263 INFO [RS_OPEN_META-regionserver/kalashnikov:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-12-12 22:06:41,267 INFO [RS_OPEN_META-regionserver/kalashnikov:0-0] wal.AbstractFSWAL(414): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=kalashnikov%2C54919%2C1544681197805.meta, suffix=.meta, logDir=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/WALs/kalashnikov,54919,1544681197805, archiveDir=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/oldWALs 2018-12-12 22:06:41,274 DEBUG [RS-EventLoopGroup-4-14] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(783): SASL client skipping handshake in unsecured configuration for addr = /127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:54906,DS-062f61d8-8f41-4a9e-a2ce-6913541fccba,DISK] 2018-12-12 22:06:41,274 DEBUG [RS-EventLoopGroup-4-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(783): SASL client skipping handshake in unsecured configuration for addr = /127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:54902,DS-dfa0bd22-3ff7-4df8-ab3c-2883ad1c4339,DISK] 2018-12-12 22:06:41,281 INFO [RS_OPEN_META-regionserver/kalashnikov:0-0] wal.AbstractFSWAL(672): New WAL /user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/WALs/kalashnikov,54919,1544681197805/kalashnikov%2C54919%2C1544681197805.meta.1544681201269.meta 2018-12-12 22:06:41,281 DEBUG [RS_OPEN_META-regionserver/kalashnikov:0-0] wal.AbstractFSWAL(762): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:54906,DS-062f61d8-8f41-4a9e-a2ce-6913541fccba,DISK], DatanodeInfoWithStorage[127.0.0.1:54902,DS-dfa0bd22-3ff7-4df8-ab3c-2883ad1c4339,DISK]] 2018-12-12 22:06:41,282 DEBUG [RS_OPEN_META-regionserver/kalashnikov:0-0] regionserver.HRegion(7177): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2018-12-12 22:06:41,324 DEBUG [RS_OPEN_META-regionserver/kalashnikov:0-0] coprocessor.CoprocessorHost(200): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2018-12-12 22:06:41,346 DEBUG [RS_OPEN_META-regionserver/kalashnikov:0-0] regionserver.HRegion(8154): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2018-12-12 22:06:41,352 INFO [RS_OPEN_META-regionserver/kalashnikov:0-0] regionserver.RegionCoprocessorHost(394): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2018-12-12 22:06:41,357 DEBUG [RS_OPEN_META-regionserver/kalashnikov:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table meta 1588230740 2018-12-12 22:06:41,357 DEBUG [RS_OPEN_META-regionserver/kalashnikov:0-0] regionserver.HRegion(833): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-12-12 22:06:41,364 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/info 2018-12-12 22:06:41,364 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/info 2018-12-12 22:06:41,365 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(237): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-12-12 22:06:41,365 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-12-12 22:06:41,366 INFO [StoreOpener-1588230740-1] regionserver.HStore(332): Store=info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2018-12-12 22:06:41,368 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/rep_barrier 2018-12-12 22:06:41,368 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/rep_barrier 2018-12-12 22:06:41,369 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(237): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-12-12 22:06:41,369 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-12-12 22:06:41,370 INFO [StoreOpener-1588230740-1] regionserver.HStore(332): Store=rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2018-12-12 22:06:41,372 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/table 2018-12-12 22:06:41,372 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/table 2018-12-12 22:06:41,372 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(237): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-12-12 22:06:41,373 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-12-12 22:06:41,374 INFO [StoreOpener-1588230740-1] regionserver.HStore(332): Store=table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2018-12-12 22:06:41,377 DEBUG [RS_OPEN_META-regionserver/kalashnikov:0-0] regionserver.HRegion(4571): Found 0 recovered edits file(s) under hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740 2018-12-12 22:06:41,380 DEBUG [RS_OPEN_META-regionserver/kalashnikov:0-0] regionserver.HRegion(4571): Found 0 recovered edits file(s) under hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/hbase/meta/1588230740 2018-12-12 22:06:41,381 DEBUG [RS_OPEN_META-regionserver/kalashnikov:0-0] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7M)) instead. 2018-12-12 22:06:41,382 DEBUG [RS_OPEN_META-regionserver/kalashnikov:0-0] regionserver.HRegion(998): writing seq id for 1588230740 2018-12-12 22:06:41,383 INFO [RS_OPEN_META-regionserver/kalashnikov:0-0] regionserver.HRegion(1002): Opened 1588230740; next sequenceid=2 2018-12-12 22:06:41,428 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2177): Post open deploy tasks for hbase:meta,,1.1588230740 2018-12-12 22:06:41,446 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=54915] assignment.RegionTransitionProcedure(290): Received report OPENED seqId=2, pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=kalashnikov,54919,1544681197805 2018-12-12 22:06:41,447 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=54915] procedure.MasterProcedureScheduler(356): Add MetaQueue(hbase:meta, xlock=false sharedLock=0 size=1) to run queue because: pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_FINISH, locked=true; AssignProcedure table=hbase:meta, region=1588230740 has lock 2018-12-12 22:06:41,447 DEBUG [PEWorker-4] procedure.MasterProcedureScheduler(366): Remove MetaQueue(hbase:meta, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_FINISH, locked=true; AssignProcedure table=hbase:meta, region=1588230740 2018-12-12 22:06:41,448 DEBUG [PEWorker-4] assignment.RegionTransitionProcedure(387): Finishing pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_FINISH, locked=true; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=kalashnikov,54919,1544681197805 2018-12-12 22:06:41,448 INFO [PEWorker-4] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as kalashnikov,54919,1544681197805 2018-12-12 22:06:41,449 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2201): Finished post open deploy task for hbase:meta,,1.1588230740 2018-12-12 22:06:41,451 DEBUG [RS_OPEN_META-regionserver/kalashnikov:0-0] handler.OpenRegionHandler(127): Opened hbase:meta,,1.1588230740 on kalashnikov,54919,1544681197805 2018-12-12 22:06:41,451 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:54915-0x167a62c89310000, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2018-12-12 22:06:41,451 DEBUG [PEWorker-4] procedure2.RootProcedureState(153): Add procedure pid=2, ppid=1, state=SUCCESS, locked=true; AssignProcedure table=hbase:meta, region=1588230740 as the 4th rollback step 2018-12-12 22:06:41,455 DEBUG [PEWorker-4] procedure.MasterProcedureScheduler(356): Add TableQueue(hbase:meta, xlock=false sharedLock=0 size=0) to run queue because: pid=2, ppid=1, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 released the shared lock 2018-12-12 22:06:41,457 DEBUG [PEWorker-4] procedure.MasterProcedureScheduler(356): Add MetaQueue(hbase:meta, xlock=false sharedLock=0 size=1) to run queue because: the exclusive lock is not held by anyone when adding pid=1, state=RUNNABLE; InitMetaProcedure table=hbase:meta 2018-12-12 22:06:41,457 INFO [PEWorker-4] procedure2.ProcedureExecutor(1897): Finished subprocedure pid=2, resume processing parent pid=1, state=RUNNABLE; InitMetaProcedure table=hbase:meta 2018-12-12 22:06:41,457 DEBUG [PEWorker-5] procedure.MasterProcedureScheduler(366): Remove MetaQueue(hbase:meta, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=1, state=RUNNABLE; InitMetaProcedure table=hbase:meta 2018-12-12 22:06:41,457 INFO [PEWorker-4] procedure2.ProcedureExecutor(1485): Finished pid=2, ppid=1, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 in 2.7060sec, unfinishedSiblingCount=0 2018-12-12 22:06:41,457 DEBUG [PEWorker-5] procedure.MasterProcedureScheduler(366): Remove TableQueue(hbase:meta, xlock=true (1) sharedLock=0 size=0) from run queue because: pid=1, state=RUNNABLE; InitMetaProcedure table=hbase:meta held the exclusive lock 2018-12-12 22:06:41,459 DEBUG [PEWorker-5] procedure2.RootProcedureState(153): Add procedure pid=1, state=SUCCESS, locked=true; InitMetaProcedure table=hbase:meta as the 5th rollback step 2018-12-12 22:06:41,463 DEBUG [PEWorker-5] procedure.MasterProcedureScheduler(356): Add TableQueue(hbase:meta, xlock=false sharedLock=0 size=0) to run queue because: pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta released the exclusive lock 2018-12-12 22:06:41,463 INFO [PEWorker-5] procedure2.ProcedureExecutor(1485): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 2.8860sec 2018-12-12 22:06:41,464 INFO [master/kalashnikov:0:becomeActiveMaster] master.HMaster(981): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1544681197848, completionTime=-1 2018-12-12 22:06:41,464 INFO [master/kalashnikov:0:becomeActiveMaster] master.ServerManager(839): Finished waiting on RegionServer count=2; waited=0ms, expected min=2 server(s), max=2 server(s), master is running 2018-12-12 22:06:41,464 DEBUG [master/kalashnikov:0:becomeActiveMaster] assignment.AssignmentManager(1201): Joining cluster... 2018-12-12 22:06:41,539 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(556): Connection from 192.168.1.137:54945, version=2.1.2, sasl=false, ugi=stack (auth:SIMPLE), service=ClientService 2018-12-12 22:06:41,583 INFO [master/kalashnikov:0:becomeActiveMaster] assignment.AssignmentManager(1213): Number of RegionServers=2 2018-12-12 22:06:41,583 INFO [master/kalashnikov:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(82): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1544681261583 2018-12-12 22:06:41,583 INFO [master/kalashnikov:0:becomeActiveMaster] assignment.AssignmentManager(1219): Joined the cluster in 119msec 2018-12-12 22:06:41,645 INFO [master/kalashnikov:0:becomeActiveMaster] master.TableNamespaceManager(96): Namespace table not found. Creating... 2018-12-12 22:06:41,648 INFO [master/kalashnikov:0:becomeActiveMaster] master.HMaster(2022): Client=null/null create 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} 2018-12-12 22:06:41,770 DEBUG [master/kalashnikov:0:becomeActiveMaster] procedure2.ProcedureExecutor(1092): Stored pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-12-12 22:06:41,770 DEBUG [master/kalashnikov:0:becomeActiveMaster] procedure.MasterProcedureScheduler(356): Add TableQueue(hbase:namespace, xlock=false sharedLock=0 size=1) to run queue because: the exclusive lock is not held by anyone when adding pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-12-12 22:06:41,772 DEBUG [PEWorker-6] procedure.MasterProcedureScheduler(366): Remove TableQueue(hbase:namespace, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-12-12 22:06:41,772 DEBUG [PEWorker-6] procedure.MasterProcedureScheduler(366): Remove TableQueue(hbase:namespace, xlock=true (3) sharedLock=0 size=0) from run queue because: pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace held the exclusive lock 2018-12-12 22:06:41,780 DEBUG [PEWorker-6] procedure2.RootProcedureState(153): Add procedure pid=3, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace as the 0th rollback step 2018-12-12 22:06:41,785 DEBUG [PEWorker-6] procedure.DeleteTableProcedure(313): Archiving region hbase:namespace,,1544681201646.b61e07ebe62dcc1570e2963289f44b29. from FS 2018-12-12 22:06:41,788 DEBUG [PEWorker-6] backup.HFileArchiver(112): ARCHIVING hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524 2018-12-12 22:06:41,791 DEBUG [PEWorker-6] backup.HFileArchiver(146): Directory hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/.tmp/data/hbase/namespace/b61e07ebe62dcc1570e2963289f44b29 empty. 2018-12-12 22:06:41,792 DEBUG [PEWorker-6] backup.HFileArchiver(461): Failed to delete directory hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/.tmp/data/hbase/namespace/b61e07ebe62dcc1570e2963289f44b29 2018-12-12 22:06:41,792 DEBUG [PEWorker-6] procedure.DeleteTableProcedure(317): Table 'hbase:namespace' archived! 2018-12-12 22:06:41,819 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54902 is added to blk_1073741833_1009{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-572a8465-16cf-4b8a-9cb7-4866d7586234:NORMAL:127.0.0.1:54906|RBW], ReplicaUC[[DISK]DS-b8b11ce4-1484-4801-8b0b-e3a38c78a69a:NORMAL:127.0.0.1:54902|RBW]]} size 0 2018-12-12 22:06:41,821 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54906 is added to blk_1073741833_1009{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b8b11ce4-1484-4801-8b0b-e3a38c78a69a:NORMAL:127.0.0.1:54902|RBW], ReplicaUC[[DISK]DS-062f61d8-8f41-4a9e-a2ce-6913541fccba:NORMAL:127.0.0.1:54906|FINALIZED]]} size 0 2018-12-12 22:06:41,823 DEBUG [PEWorker-6] util.FSTableDescriptors(684): Wrote into hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2018-12-12 22:06:41,826 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(7003): creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/.tmp Table name == hbase:namespace 2018-12-12 22:06:41,842 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54902 is added to blk_1073741834_1010{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-062f61d8-8f41-4a9e-a2ce-6913541fccba:NORMAL:127.0.0.1:54906|RBW], ReplicaUC[[DISK]DS-dfa0bd22-3ff7-4df8-ab3c-2883ad1c4339:NORMAL:127.0.0.1:54902|RBW]]} size 0 2018-12-12 22:06:41,843 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54906 is added to blk_1073741834_1010{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dfa0bd22-3ff7-4df8-ab3c-2883ad1c4339:NORMAL:127.0.0.1:54902|RBW], ReplicaUC[[DISK]DS-572a8465-16cf-4b8a-9cb7-4866d7586234:NORMAL:127.0.0.1:54906|FINALIZED]]} size 0 2018-12-12 22:06:41,845 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(833): Instantiated hbase:namespace,,1544681201646.b61e07ebe62dcc1570e2963289f44b29.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-12-12 22:06:41,845 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1541): Closing b61e07ebe62dcc1570e2963289f44b29, disabling compactions & flushes 2018-12-12 22:06:41,845 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1581): Updates disabled for region hbase:namespace,,1544681201646.b61e07ebe62dcc1570e2963289f44b29. 2018-12-12 22:06:41,845 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1698): Closed hbase:namespace,,1544681201646.b61e07ebe62dcc1570e2963289f44b29. 2018-12-12 22:06:41,848 DEBUG [PEWorker-6] procedure2.RootProcedureState(153): Add procedure pid=3, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace as the 1th rollback step 2018-12-12 22:06:41,890 DEBUG [PEWorker-6] hbase.MetaTableAccessor(2153): Put {"totalColumns":2,"row":"hbase:namespace,,1544681201646.b61e07ebe62dcc1570e2963289f44b29.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":1544681201852},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1544681201852}]},"ts":1544681201852} 2018-12-12 22:06:41,930 INFO [PEWorker-6] hbase.MetaTableAccessor(1528): Added 1 regions to meta. 2018-12-12 22:06:41,930 DEBUG [PEWorker-6] procedure2.RootProcedureState(153): Add procedure pid=3, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace as the 2th rollback step 2018-12-12 22:06:41,937 DEBUG [PEWorker-6] hbase.MetaTableAccessor(2153): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1544681201933}]},"ts":1544681201933} 2018-12-12 22:06:41,942 INFO [PEWorker-6] hbase.MetaTableAccessor(1673): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2018-12-12 22:06:41,953 INFO [PEWorker-6] procedure2.ProcedureExecutor(1758): Initialized subprocedures=[{pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=b61e07ebe62dcc1570e2963289f44b29, target=kalashnikov,54919,1544681197805}] 2018-12-12 22:06:41,953 DEBUG [PEWorker-6] procedure2.RootProcedureState(153): Add procedure pid=3, state=WAITING:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace as the 3th rollback step 2018-12-12 22:06:41,955 DEBUG [PEWorker-6] procedure.MasterProcedureScheduler(356): Add TableQueue(hbase:namespace, xlock=true (3) sharedLock=0 size=1) to run queue because: pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=b61e07ebe62dcc1570e2963289f44b29, target=kalashnikov,54919,1544681197805 has the excusive lock access 2018-12-12 22:06:41,955 DEBUG [PEWorker-8] procedure.MasterProcedureScheduler(366): Remove TableQueue(hbase:namespace, xlock=true (3) sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=b61e07ebe62dcc1570e2963289f44b29, target=kalashnikov,54919,1544681197805 2018-12-12 22:06:41,956 INFO [PEWorker-8] procedure.MasterProcedureScheduler(741): Took xlock for pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=b61e07ebe62dcc1570e2963289f44b29, target=kalashnikov,54919,1544681197805 2018-12-12 22:06:42,067 DEBUG [PEWorker-6] procedure.MasterProcedureScheduler(356): Add TableQueue(hbase:namespace, xlock=false sharedLock=1 size=0) to run queue because: pid=3, state=WAITING:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace released the exclusive lock 2018-12-12 22:06:42,179 INFO [PEWorker-8] assignment.AssignProcedure(254): Starting pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure table=hbase:namespace, region=b61e07ebe62dcc1570e2963289f44b29, target=kalashnikov,54919,1544681197805; rit=OFFLINE, location=kalashnikov,54919,1544681197805; forceNewPlan=false, retain=false 2018-12-12 22:06:42,179 DEBUG [PEWorker-8] procedure2.RootProcedureState(153): Add procedure pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:namespace, region=b61e07ebe62dcc1570e2963289f44b29, target=kalashnikov,54919,1544681197805 as the 4th rollback step 2018-12-12 22:06:42,337 INFO [master/kalashnikov:0] balancer.BaseLoadBalancer(1531): Reassigned 1 regions. 1 retained the pre-restart assignment. 2018-12-12 22:06:42,337 DEBUG [master/kalashnikov:0] procedure.MasterProcedureScheduler(356): Add TableQueue(hbase:namespace, xlock=false sharedLock=1 size=1) to run queue because: pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:namespace, region=b61e07ebe62dcc1570e2963289f44b29, target=kalashnikov,54919,1544681197805 has lock 2018-12-12 22:06:42,338 DEBUG [PEWorker-10] procedure.MasterProcedureScheduler(366): Remove TableQueue(hbase:namespace, xlock=false sharedLock=1 size=0) from run queue because: queue is empty after polling out pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:namespace, region=b61e07ebe62dcc1570e2963289f44b29, target=kalashnikov,54919,1544681197805 2018-12-12 22:06:42,338 INFO [PEWorker-10] assignment.RegionStateStore(200): pid=4 updating hbase:meta row=b61e07ebe62dcc1570e2963289f44b29, regionState=OPENING, regionLocation=kalashnikov,54919,1544681197805 2018-12-12 22:06:42,344 INFO [PEWorker-10] assignment.RegionTransitionProcedure(267): Dispatch pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:namespace, region=b61e07ebe62dcc1570e2963289f44b29, target=kalashnikov,54919,1544681197805 2018-12-12 22:06:42,344 DEBUG [PEWorker-10] procedure2.RootProcedureState(153): Add procedure pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:namespace, region=b61e07ebe62dcc1570e2963289f44b29, target=kalashnikov,54919,1544681197805 as the 5th rollback step 2018-12-12 22:06:42,504 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=54919] regionserver.RSRpcServices(1987): Open hbase:namespace,,1544681201646.b61e07ebe62dcc1570e2963289f44b29. 2018-12-12 22:06:42,513 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/kalashnikov:0-0] regionserver.HRegion(7177): Opening region: {ENCODED => b61e07ebe62dcc1570e2963289f44b29, NAME => 'hbase:namespace,,1544681201646.b61e07ebe62dcc1570e2963289f44b29.', STARTKEY => '', ENDKEY => ''} 2018-12-12 22:06:42,515 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/kalashnikov:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table namespace b61e07ebe62dcc1570e2963289f44b29 2018-12-12 22:06:42,515 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/kalashnikov:0-0] regionserver.HRegion(833): Instantiated hbase:namespace,,1544681201646.b61e07ebe62dcc1570e2963289f44b29.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-12-12 22:06:42,520 DEBUG [StoreOpener-b61e07ebe62dcc1570e2963289f44b29-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/namespace/b61e07ebe62dcc1570e2963289f44b29/info 2018-12-12 22:06:42,520 DEBUG [StoreOpener-b61e07ebe62dcc1570e2963289f44b29-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/namespace/b61e07ebe62dcc1570e2963289f44b29/info 2018-12-12 22:06:42,521 INFO [StoreOpener-b61e07ebe62dcc1570e2963289f44b29-1] hfile.CacheConfig(237): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-12-12 22:06:42,521 INFO [StoreOpener-b61e07ebe62dcc1570e2963289f44b29-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-12-12 22:06:42,522 INFO [StoreOpener-b61e07ebe62dcc1570e2963289f44b29-1] regionserver.HStore(332): Store=info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2018-12-12 22:06:42,524 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/kalashnikov:0-0] regionserver.HRegion(4571): Found 0 recovered edits file(s) under hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/namespace/b61e07ebe62dcc1570e2963289f44b29 2018-12-12 22:06:42,525 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/kalashnikov:0-0] regionserver.HRegion(4571): Found 0 recovered edits file(s) under hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/hbase/namespace/b61e07ebe62dcc1570e2963289f44b29 2018-12-12 22:06:42,528 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/kalashnikov:0-0] regionserver.HRegion(998): writing seq id for b61e07ebe62dcc1570e2963289f44b29 2018-12-12 22:06:42,532 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/kalashnikov:0-0] wal.WALSplitter(695): Wrote file=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/hbase/namespace/b61e07ebe62dcc1570e2963289f44b29/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-12-12 22:06:42,532 INFO [RS_OPEN_PRIORITY_REGION-regionserver/kalashnikov:0-0] regionserver.HRegion(1002): Opened b61e07ebe62dcc1570e2963289f44b29; next sequenceid=2 2018-12-12 22:06:42,535 INFO [PostOpenDeployTasks:b61e07ebe62dcc1570e2963289f44b29] regionserver.HRegionServer(2177): Post open deploy tasks for hbase:namespace,,1544681201646.b61e07ebe62dcc1570e2963289f44b29. 2018-12-12 22:06:42,536 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=54915] assignment.RegionTransitionProcedure(290): Received report OPENED seqId=2, pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:namespace, region=b61e07ebe62dcc1570e2963289f44b29, target=kalashnikov,54919,1544681197805; rit=OPENING, location=kalashnikov,54919,1544681197805 2018-12-12 22:06:42,537 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=54915] procedure.MasterProcedureScheduler(356): Add TableQueue(hbase:namespace, xlock=false sharedLock=1 size=1) to run queue because: pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_FINISH, locked=true; AssignProcedure table=hbase:namespace, region=b61e07ebe62dcc1570e2963289f44b29, target=kalashnikov,54919,1544681197805 has lock 2018-12-12 22:06:42,537 DEBUG [PEWorker-11] procedure.MasterProcedureScheduler(366): Remove TableQueue(hbase:namespace, xlock=false sharedLock=1 size=0) from run queue because: queue is empty after polling out pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_FINISH, locked=true; AssignProcedure table=hbase:namespace, region=b61e07ebe62dcc1570e2963289f44b29, target=kalashnikov,54919,1544681197805 2018-12-12 22:06:42,537 DEBUG [PEWorker-11] assignment.RegionTransitionProcedure(387): Finishing pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_FINISH, locked=true; AssignProcedure table=hbase:namespace, region=b61e07ebe62dcc1570e2963289f44b29, target=kalashnikov,54919,1544681197805; rit=OPENING, location=kalashnikov,54919,1544681197805 2018-12-12 22:06:42,538 DEBUG [PostOpenDeployTasks:b61e07ebe62dcc1570e2963289f44b29] regionserver.HRegionServer(2201): Finished post open deploy task for hbase:namespace,,1544681201646.b61e07ebe62dcc1570e2963289f44b29. 2018-12-12 22:06:42,538 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/kalashnikov:0-0] handler.OpenRegionHandler(127): Opened hbase:namespace,,1544681201646.b61e07ebe62dcc1570e2963289f44b29. on kalashnikov,54919,1544681197805 2018-12-12 22:06:42,539 INFO [PEWorker-11] assignment.RegionStateStore(200): pid=4 updating hbase:meta row=b61e07ebe62dcc1570e2963289f44b29, regionState=OPEN, openSeqNum=2, regionLocation=kalashnikov,54919,1544681197805 2018-12-12 22:06:42,544 DEBUG [PEWorker-11] procedure2.RootProcedureState(153): Add procedure pid=4, ppid=3, state=SUCCESS, locked=true; AssignProcedure table=hbase:namespace, region=b61e07ebe62dcc1570e2963289f44b29, target=kalashnikov,54919,1544681197805 as the 6th rollback step 2018-12-12 22:06:42,554 DEBUG [PEWorker-11] procedure.MasterProcedureScheduler(356): Add TableQueue(hbase:namespace, xlock=false sharedLock=0 size=0) to run queue because: pid=4, ppid=3, state=SUCCESS; AssignProcedure table=hbase:namespace, region=b61e07ebe62dcc1570e2963289f44b29, target=kalashnikov,54919,1544681197805 released the shared lock 2018-12-12 22:06:42,556 DEBUG [PEWorker-11] procedure.MasterProcedureScheduler(356): Add TableQueue(hbase:namespace, xlock=false sharedLock=0 size=1) to run queue because: the exclusive lock is not held by anyone when adding pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace 2018-12-12 22:06:42,556 INFO [PEWorker-11] procedure2.ProcedureExecutor(1897): Finished subprocedure pid=4, resume processing parent pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace 2018-12-12 22:06:42,556 DEBUG [PEWorker-12] procedure.MasterProcedureScheduler(366): Remove TableQueue(hbase:namespace, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace 2018-12-12 22:06:42,556 INFO [PEWorker-11] procedure2.ProcedureExecutor(1485): Finished pid=4, ppid=3, state=SUCCESS; AssignProcedure table=hbase:namespace, region=b61e07ebe62dcc1570e2963289f44b29, target=kalashnikov,54919,1544681197805 in 591msec, unfinishedSiblingCount=0 2018-12-12 22:06:42,557 DEBUG [PEWorker-12] procedure.MasterProcedureScheduler(366): Remove TableQueue(hbase:namespace, xlock=true (3) sharedLock=0 size=0) from run queue because: pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace held the exclusive lock 2018-12-12 22:06:42,559 DEBUG [PEWorker-12] hbase.MetaTableAccessor(2153): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1544681202559}]},"ts":1544681202559} 2018-12-12 22:06:42,563 INFO [PEWorker-12] hbase.MetaTableAccessor(1673): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2018-12-12 22:06:42,564 DEBUG [PEWorker-12] procedure2.RootProcedureState(153): Add procedure pid=3, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace as the 7th rollback step 2018-12-12 22:06:42,566 DEBUG [PEWorker-12] procedure2.RootProcedureState(153): Add procedure pid=3, state=SUCCESS, locked=true; CreateTableProcedure table=hbase:namespace as the 8th rollback step 2018-12-12 22:06:42,570 DEBUG [PEWorker-12] procedure.MasterProcedureScheduler(356): Add TableQueue(hbase:namespace, xlock=false sharedLock=0 size=0) to run queue because: pid=3, state=SUCCESS; CreateTableProcedure table=hbase:namespace released the exclusive lock 2018-12-12 22:06:42,570 INFO [PEWorker-12] procedure2.ProcedureExecutor(1485): Finished pid=3, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 917msec 2018-12-12 22:06:42,579 DEBUG [master/kalashnikov:0:becomeActiveMaster] zookeeper.ZKUtil(357): master:54915-0x167a62c89310000, quorum=localhost:63470, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2018-12-12 22:06:42,580 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:54915-0x167a62c89310000, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2018-12-12 22:06:42,733 DEBUG [master/kalashnikov:0:becomeActiveMaster] procedure2.ProcedureExecutor(1092): Stored pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-12-12 22:06:42,733 DEBUG [master/kalashnikov:0:becomeActiveMaster] procedure.MasterProcedureScheduler(356): Add TableQueue(hbase:namespace, xlock=false sharedLock=0 size=1) to run queue because: the exclusive lock is not held by anyone when adding pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-12-12 22:06:42,734 DEBUG [PEWorker-14] procedure.MasterProcedureScheduler(366): Remove TableQueue(hbase:namespace, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-12-12 22:06:42,740 DEBUG [PEWorker-14] procedure2.RootProcedureState(153): Add procedure pid=5, state=RUNNABLE:CREATE_NAMESPACE_CREATE_DIRECTORY, locked=true; CreateNamespaceProcedure, namespace=default as the 0th rollback step 2018-12-12 22:06:42,742 DEBUG [PEWorker-14] procedure2.RootProcedureState(153): Add procedure pid=5, state=RUNNABLE:CREATE_NAMESPACE_INSERT_INTO_NS_TABLE, locked=true; CreateNamespaceProcedure, namespace=default as the 1th rollback step 2018-12-12 22:06:42,748 DEBUG [PEWorker-14] procedure2.RootProcedureState(153): Add procedure pid=5, state=RUNNABLE:CREATE_NAMESPACE_UPDATE_ZK, locked=true; CreateNamespaceProcedure, namespace=default as the 2th rollback step 2018-12-12 22:06:42,750 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:54915-0x167a62c89310000, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-12-12 22:06:42,752 DEBUG [PEWorker-14] procedure2.RootProcedureState(153): Add procedure pid=5, state=RUNNABLE:CREATE_NAMESPACE_SET_NAMESPACE_QUOTA, locked=true; CreateNamespaceProcedure, namespace=default as the 3th rollback step 2018-12-12 22:06:42,753 DEBUG [PEWorker-14] procedure2.RootProcedureState(153): Add procedure pid=5, state=SUCCESS, locked=true; CreateNamespaceProcedure, namespace=default as the 4th rollback step 2018-12-12 22:06:42,756 DEBUG [PEWorker-14] procedure.MasterProcedureScheduler(356): Add TableQueue(hbase:namespace, xlock=false sharedLock=0 size=0) to run queue because: pid=5, state=SUCCESS; CreateNamespaceProcedure, namespace=default released namespace exclusive lock 2018-12-12 22:06:42,756 INFO [PEWorker-14] procedure2.ProcedureExecutor(1485): Finished pid=5, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 142msec 2018-12-12 22:06:42,880 DEBUG [master/kalashnikov:0:becomeActiveMaster] procedure2.ProcedureExecutor(1092): Stored pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-12-12 22:06:42,880 DEBUG [master/kalashnikov:0:becomeActiveMaster] procedure.MasterProcedureScheduler(356): Add TableQueue(hbase:namespace, xlock=false sharedLock=0 size=1) to run queue because: the exclusive lock is not held by anyone when adding pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-12-12 22:06:42,881 DEBUG [PEWorker-15] procedure.MasterProcedureScheduler(366): Remove TableQueue(hbase:namespace, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-12-12 22:06:42,885 DEBUG [PEWorker-15] procedure2.RootProcedureState(153): Add procedure pid=6, state=RUNNABLE:CREATE_NAMESPACE_CREATE_DIRECTORY, locked=true; CreateNamespaceProcedure, namespace=hbase as the 0th rollback step 2018-12-12 22:06:42,888 DEBUG [PEWorker-15] procedure2.RootProcedureState(153): Add procedure pid=6, state=RUNNABLE:CREATE_NAMESPACE_INSERT_INTO_NS_TABLE, locked=true; CreateNamespaceProcedure, namespace=hbase as the 1th rollback step 2018-12-12 22:06:42,893 DEBUG [PEWorker-15] procedure2.RootProcedureState(153): Add procedure pid=6, state=RUNNABLE:CREATE_NAMESPACE_UPDATE_ZK, locked=true; CreateNamespaceProcedure, namespace=hbase as the 2th rollback step 2018-12-12 22:06:42,896 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:54915-0x167a62c89310000, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-12-12 22:06:42,897 DEBUG [PEWorker-15] procedure2.RootProcedureState(153): Add procedure pid=6, state=RUNNABLE:CREATE_NAMESPACE_SET_NAMESPACE_QUOTA, locked=true; CreateNamespaceProcedure, namespace=hbase as the 3th rollback step 2018-12-12 22:06:42,898 DEBUG [PEWorker-15] procedure2.RootProcedureState(153): Add procedure pid=6, state=SUCCESS, locked=true; CreateNamespaceProcedure, namespace=hbase as the 4th rollback step 2018-12-12 22:06:42,901 DEBUG [PEWorker-15] procedure.MasterProcedureScheduler(356): Add TableQueue(hbase:namespace, xlock=false sharedLock=0 size=0) to run queue because: pid=6, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase released namespace exclusive lock 2018-12-12 22:06:42,901 INFO [PEWorker-15] procedure2.ProcedureExecutor(1485): Finished pid=6, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 130msec 2018-12-12 22:06:42,918 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:54915-0x167a62c89310000, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2018-12-12 22:06:42,920 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:54915-0x167a62c89310000, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2018-12-12 22:06:42,920 INFO [master/kalashnikov:0:becomeActiveMaster] master.HMaster(1049): Master has completed initialization 5.060sec 2018-12-12 22:06:42,925 INFO [master/kalashnikov:0:becomeActiveMaster] quotas.MasterQuotaManager(80): Quota support disabled 2018-12-12 22:06:42,925 INFO [master/kalashnikov:0:becomeActiveMaster] zookeeper.ZKWatcher(205): not a secure deployment, proceeding 2018-12-12 22:06:42,931 INFO [master/kalashnikov:0:becomeActiveMaster] balancer.RegionLocationFinder(308): Refreshing block distribution cache for 2 regions (Can take a while on big cluster) 2018-12-12 22:06:42,936 INFO [master/kalashnikov:0:becomeActiveMaster] balancer.RegionLocationFinder(324): Finished refreshing block distribution cache for 2 regions 2018-12-12 22:06:42,936 DEBUG [master/kalashnikov:0:becomeActiveMaster] master.HMaster(1113): Balancer post startup initialization complete, took 0 seconds 2018-12-12 22:06:42,949 INFO [Time-limited test] zookeeper.ReadOnlyZKClient(139): Connect 0x1f41fb2a to localhost:63470 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2018-12-12 22:06:42,953 DEBUG [Time-limited test] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6c3efba3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-12-12 22:06:42,961 INFO [RS-EventLoopGroup-4-10] ipc.ServerRpcConnection(556): Connection from 192.168.1.137:54953, version=2.1.2, sasl=false, ugi=stack (auth:SIMPLE), service=ClientService 2018-12-12 22:06:42,971 INFO [Time-limited test] hbase.HBaseTestingUtility(1052): Minicluster is up; activeMaster=kalashnikov,54915,1544681196595 2018-12-12 22:06:43,133 INFO [Time-limited test] hbase.ResourceChecker(148): before: regionserver.TestCleanupMetaWAL#testCleanupMetaWAL Thread=299, OpenFileDescriptor=832, MaxFileDescriptor=10240, SystemLoadAverage=305, ProcessCount=483, AvailableMemoryMB=725 2018-12-12 22:06:43,150 INFO [RS-EventLoopGroup-1-4] ipc.ServerRpcConnection(556): Connection from 192.168.1.137:54954, version=2.1.2, sasl=false, ugi=stack (auth:SIMPLE), service=MasterService 2018-12-12 22:06:43,163 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54915] master.HMaster$4(1986): Client=stack//192.168.1.137 create 'test', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}, {NAME => 'cf', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2018-12-12 22:06:43,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54915] procedure2.ProcedureExecutor(1092): Stored pid=7, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=test 2018-12-12 22:06:43,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54915] procedure.MasterProcedureScheduler(356): Add TableQueue(test, xlock=false sharedLock=0 size=1) to run queue because: the exclusive lock is not held by anyone when adding pid=7, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=test 2018-12-12 22:06:43,289 DEBUG [PEWorker-9] procedure.MasterProcedureScheduler(366): Remove TableQueue(test, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=7, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=test 2018-12-12 22:06:43,290 DEBUG [PEWorker-9] procedure.MasterProcedureScheduler(366): Remove TableQueue(test, xlock=true (7) sharedLock=0 size=0) from run queue because: pid=7, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=test held the exclusive lock 2018-12-12 22:06:43,296 DEBUG [PEWorker-9] procedure2.RootProcedureState(153): Add procedure pid=7, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=test as the 0th rollback step 2018-12-12 22:06:43,299 DEBUG [PEWorker-9] procedure.DeleteTableProcedure(313): Archiving region test,,1544681203160.30eb0f9efacc61d337395de976dcb330. from FS 2018-12-12 22:06:43,300 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54915] master.MasterRpcServices(631): Client=stack//192.168.1.137 procedure request for creating table: namespace: "default" qualifier: "test" procId is: 7 2018-12-12 22:06:43,300 DEBUG [PEWorker-9] backup.HFileArchiver(112): ARCHIVING hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524 2018-12-12 22:06:43,301 DEBUG [PEWorker-9] backup.HFileArchiver(146): Directory hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/.tmp/data/default/test/30eb0f9efacc61d337395de976dcb330 empty. 2018-12-12 22:06:43,302 DEBUG [PEWorker-9] backup.HFileArchiver(461): Failed to delete directory hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/.tmp/data/default/test/30eb0f9efacc61d337395de976dcb330 2018-12-12 22:06:43,303 DEBUG [PEWorker-9] procedure.DeleteTableProcedure(317): Table 'test' archived! 2018-12-12 22:06:43,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54915] master.MasterRpcServices(1179): Checking to see if procedure is done pid=7 2018-12-12 22:06:43,337 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54902 is added to blk_1073741835_1011{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-062f61d8-8f41-4a9e-a2ce-6913541fccba:NORMAL:127.0.0.1:54906|RBW], ReplicaUC[[DISK]DS-b8b11ce4-1484-4801-8b0b-e3a38c78a69a:NORMAL:127.0.0.1:54902|FINALIZED]]} size 0 2018-12-12 22:06:43,340 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54906 is added to blk_1073741835_1011{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-062f61d8-8f41-4a9e-a2ce-6913541fccba:NORMAL:127.0.0.1:54906|RBW], ReplicaUC[[DISK]DS-b8b11ce4-1484-4801-8b0b-e3a38c78a69a:NORMAL:127.0.0.1:54902|FINALIZED]]} size 0 2018-12-12 22:06:43,344 DEBUG [PEWorker-9] util.FSTableDescriptors(684): Wrote into hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/.tmp/data/default/test/.tabledesc/.tableinfo.0000000001 2018-12-12 22:06:43,346 INFO [RegionOpenAndInitThread-test-1] regionserver.HRegion(7003): creating HRegion test HTD == 'test', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}, {NAME => 'cf', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} RootDir = hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/.tmp Table name == test 2018-12-12 22:06:43,367 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54902 is added to blk_1073741836_1012{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-062f61d8-8f41-4a9e-a2ce-6913541fccba:NORMAL:127.0.0.1:54906|RBW], ReplicaUC[[DISK]DS-dfa0bd22-3ff7-4df8-ab3c-2883ad1c4339:NORMAL:127.0.0.1:54902|FINALIZED]]} size 0 2018-12-12 22:06:43,368 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54906 is added to blk_1073741836_1012 size 39 2018-12-12 22:06:43,369 DEBUG [RegionOpenAndInitThread-test-1] regionserver.HRegion(833): Instantiated test,,1544681203160.30eb0f9efacc61d337395de976dcb330.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-12-12 22:06:43,370 DEBUG [RegionOpenAndInitThread-test-1] regionserver.HRegion(1541): Closing 30eb0f9efacc61d337395de976dcb330, disabling compactions & flushes 2018-12-12 22:06:43,370 DEBUG [RegionOpenAndInitThread-test-1] regionserver.HRegion(1581): Updates disabled for region test,,1544681203160.30eb0f9efacc61d337395de976dcb330. 2018-12-12 22:06:43,370 INFO [RegionOpenAndInitThread-test-1] regionserver.HRegion(1698): Closed test,,1544681203160.30eb0f9efacc61d337395de976dcb330. 2018-12-12 22:06:43,375 DEBUG [PEWorker-9] procedure2.RootProcedureState(153): Add procedure pid=7, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=test as the 1th rollback step 2018-12-12 22:06:43,379 DEBUG [PEWorker-9] hbase.MetaTableAccessor(2153): Put {"totalColumns":2,"row":"test,,1544681203160.30eb0f9efacc61d337395de976dcb330.","families":{"info":[{"qualifier":"regioninfo","vlen":38,"tag":[],"timestamp":1544681203378},{"qualifier":"state","vlen":6,"tag":[],"timestamp":1544681203378}]},"ts":1544681203378} 2018-12-12 22:06:43,382 INFO [PEWorker-9] hbase.MetaTableAccessor(1528): Added 1 regions to meta. 2018-12-12 22:06:43,382 DEBUG [PEWorker-9] procedure2.RootProcedureState(153): Add procedure pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=test as the 2th rollback step 2018-12-12 22:06:43,384 DEBUG [PEWorker-9] hbase.MetaTableAccessor(2153): Put {"totalColumns":1,"row":"test","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1544681203384}]},"ts":1544681203384} 2018-12-12 22:06:43,393 INFO [PEWorker-9] hbase.MetaTableAccessor(1673): Updated tableName=test, state=ENABLING in hbase:meta 2018-12-12 22:06:43,406 INFO [PEWorker-9] procedure2.ProcedureExecutor(1758): Initialized subprocedures=[{pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=test, region=30eb0f9efacc61d337395de976dcb330, target=kalashnikov,54919,1544681197805}] 2018-12-12 22:06:43,406 DEBUG [PEWorker-9] procedure2.RootProcedureState(153): Add procedure pid=7, state=WAITING:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=test as the 3th rollback step 2018-12-12 22:06:43,409 DEBUG [PEWorker-9] procedure.MasterProcedureScheduler(356): Add TableQueue(test, xlock=true (7) sharedLock=0 size=1) to run queue because: pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=test, region=30eb0f9efacc61d337395de976dcb330, target=kalashnikov,54919,1544681197805 has the excusive lock access 2018-12-12 22:06:43,410 DEBUG [PEWorker-7] procedure.MasterProcedureScheduler(366): Remove TableQueue(test, xlock=true (7) sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=test, region=30eb0f9efacc61d337395de976dcb330, target=kalashnikov,54919,1544681197805 2018-12-12 22:06:43,411 INFO [PEWorker-7] procedure.MasterProcedureScheduler(741): Took xlock for pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=test, region=30eb0f9efacc61d337395de976dcb330, target=kalashnikov,54919,1544681197805 2018-12-12 22:06:43,413 DEBUG [PEWorker-9] procedure.MasterProcedureScheduler(356): Add TableQueue(test, xlock=false sharedLock=1 size=0) to run queue because: pid=7, state=WAITING:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=test released the exclusive lock 2018-12-12 22:06:43,413 INFO [PEWorker-7] assignment.AssignProcedure(254): Starting pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure table=test, region=30eb0f9efacc61d337395de976dcb330, target=kalashnikov,54919,1544681197805; rit=OFFLINE, location=kalashnikov,54919,1544681197805; forceNewPlan=false, retain=false 2018-12-12 22:06:43,413 DEBUG [PEWorker-7] procedure2.RootProcedureState(153): Add procedure pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=test, region=30eb0f9efacc61d337395de976dcb330, target=kalashnikov,54919,1544681197805 as the 4th rollback step 2018-12-12 22:06:43,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54915] master.MasterRpcServices(1179): Checking to see if procedure is done pid=7 2018-12-12 22:06:43,566 INFO [master/kalashnikov:0] balancer.BaseLoadBalancer(1531): Reassigned 1 regions. 1 retained the pre-restart assignment. 2018-12-12 22:06:43,567 DEBUG [master/kalashnikov:0] procedure.MasterProcedureScheduler(356): Add TableQueue(test, xlock=false sharedLock=1 size=1) to run queue because: pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=test, region=30eb0f9efacc61d337395de976dcb330, target=kalashnikov,54919,1544681197805 has lock 2018-12-12 22:06:43,567 DEBUG [PEWorker-2] procedure.MasterProcedureScheduler(366): Remove TableQueue(test, xlock=false sharedLock=1 size=0) from run queue because: queue is empty after polling out pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=test, region=30eb0f9efacc61d337395de976dcb330, target=kalashnikov,54919,1544681197805 2018-12-12 22:06:43,567 INFO [PEWorker-2] assignment.RegionStateStore(200): pid=8 updating hbase:meta row=30eb0f9efacc61d337395de976dcb330, regionState=OPENING, regionLocation=kalashnikov,54919,1544681197805 2018-12-12 22:06:43,627 INFO [PEWorker-2] assignment.RegionTransitionProcedure(267): Dispatch pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=test, region=30eb0f9efacc61d337395de976dcb330, target=kalashnikov,54919,1544681197805 2018-12-12 22:06:43,628 DEBUG [PEWorker-2] procedure2.RootProcedureState(153): Add procedure pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=test, region=30eb0f9efacc61d337395de976dcb330, target=kalashnikov,54919,1544681197805 as the 5th rollback step 2018-12-12 22:06:43,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54915] master.MasterRpcServices(1179): Checking to see if procedure is done pid=7 2018-12-12 22:06:43,781 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=54919] regionserver.RSRpcServices(1987): Open test,,1544681203160.30eb0f9efacc61d337395de976dcb330. 2018-12-12 22:06:43,785 DEBUG [RS_OPEN_REGION-regionserver/kalashnikov:0-0] regionserver.HRegion(7177): Opening region: {ENCODED => 30eb0f9efacc61d337395de976dcb330, NAME => 'test,,1544681203160.30eb0f9efacc61d337395de976dcb330.', STARTKEY => '', ENDKEY => ''} 2018-12-12 22:06:43,785 DEBUG [RS_OPEN_REGION-regionserver/kalashnikov:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table test 30eb0f9efacc61d337395de976dcb330 2018-12-12 22:06:43,785 DEBUG [RS_OPEN_REGION-regionserver/kalashnikov:0-0] regionserver.HRegion(833): Instantiated test,,1544681203160.30eb0f9efacc61d337395de976dcb330.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-12-12 22:06:43,791 DEBUG [StoreOpener-30eb0f9efacc61d337395de976dcb330-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/default/test/30eb0f9efacc61d337395de976dcb330/cf 2018-12-12 22:06:43,791 DEBUG [StoreOpener-30eb0f9efacc61d337395de976dcb330-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/default/test/30eb0f9efacc61d337395de976dcb330/cf 2018-12-12 22:06:43,796 INFO [StoreOpener-30eb0f9efacc61d337395de976dcb330-1] hfile.CacheConfig(237): Created cacheConfig for cf: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-12-12 22:06:43,797 INFO [StoreOpener-30eb0f9efacc61d337395de976dcb330-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-12-12 22:06:43,798 INFO [StoreOpener-30eb0f9efacc61d337395de976dcb330-1] regionserver.HStore(332): Store=cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2018-12-12 22:06:43,799 DEBUG [RS_OPEN_REGION-regionserver/kalashnikov:0-0] regionserver.HRegion(4571): Found 0 recovered edits file(s) under hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/default/test/30eb0f9efacc61d337395de976dcb330 2018-12-12 22:06:43,801 DEBUG [RS_OPEN_REGION-regionserver/kalashnikov:0-0] regionserver.HRegion(4571): Found 0 recovered edits file(s) under hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/default/test/30eb0f9efacc61d337395de976dcb330 2018-12-12 22:06:43,804 DEBUG [RS_OPEN_REGION-regionserver/kalashnikov:0-0] regionserver.HRegion(998): writing seq id for 30eb0f9efacc61d337395de976dcb330 2018-12-12 22:06:43,808 DEBUG [RS_OPEN_REGION-regionserver/kalashnikov:0-0] wal.WALSplitter(695): Wrote file=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/default/test/30eb0f9efacc61d337395de976dcb330/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2018-12-12 22:06:43,809 INFO [RS_OPEN_REGION-regionserver/kalashnikov:0-0] regionserver.HRegion(1002): Opened 30eb0f9efacc61d337395de976dcb330; next sequenceid=2 2018-12-12 22:06:43,810 INFO [PostOpenDeployTasks:30eb0f9efacc61d337395de976dcb330] regionserver.HRegionServer(2177): Post open deploy tasks for test,,1544681203160.30eb0f9efacc61d337395de976dcb330. 2018-12-12 22:06:43,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54915] assignment.RegionTransitionProcedure(290): Received report OPENED seqId=2, pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=test, region=30eb0f9efacc61d337395de976dcb330, target=kalashnikov,54919,1544681197805; rit=OPENING, location=kalashnikov,54919,1544681197805 2018-12-12 22:06:43,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54915] procedure.MasterProcedureScheduler(356): Add TableQueue(test, xlock=false sharedLock=1 size=1) to run queue because: pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH, locked=true; AssignProcedure table=test, region=30eb0f9efacc61d337395de976dcb330, target=kalashnikov,54919,1544681197805 has lock 2018-12-12 22:06:43,812 DEBUG [PEWorker-4] procedure.MasterProcedureScheduler(366): Remove TableQueue(test, xlock=false sharedLock=1 size=0) from run queue because: queue is empty after polling out pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH, locked=true; AssignProcedure table=test, region=30eb0f9efacc61d337395de976dcb330, target=kalashnikov,54919,1544681197805 2018-12-12 22:06:43,812 DEBUG [PEWorker-4] assignment.RegionTransitionProcedure(387): Finishing pid=8, ppid=7, state=RUNNABLE:REGION_TRANSITION_FINISH, locked=true; AssignProcedure table=test, region=30eb0f9efacc61d337395de976dcb330, target=kalashnikov,54919,1544681197805; rit=OPENING, location=kalashnikov,54919,1544681197805 2018-12-12 22:06:43,812 DEBUG [PostOpenDeployTasks:30eb0f9efacc61d337395de976dcb330] regionserver.HRegionServer(2201): Finished post open deploy task for test,,1544681203160.30eb0f9efacc61d337395de976dcb330. 2018-12-12 22:06:43,812 INFO [PEWorker-4] assignment.RegionStateStore(200): pid=8 updating hbase:meta row=30eb0f9efacc61d337395de976dcb330, regionState=OPEN, openSeqNum=2, regionLocation=kalashnikov,54919,1544681197805 2018-12-12 22:06:43,813 DEBUG [RS_OPEN_REGION-regionserver/kalashnikov:0-0] handler.OpenRegionHandler(127): Opened test,,1544681203160.30eb0f9efacc61d337395de976dcb330. on kalashnikov,54919,1544681197805 2018-12-12 22:06:43,816 DEBUG [PEWorker-4] procedure2.RootProcedureState(153): Add procedure pid=8, ppid=7, state=SUCCESS, locked=true; AssignProcedure table=test, region=30eb0f9efacc61d337395de976dcb330, target=kalashnikov,54919,1544681197805 as the 6th rollback step 2018-12-12 22:06:43,819 DEBUG [PEWorker-4] procedure.MasterProcedureScheduler(356): Add TableQueue(test, xlock=false sharedLock=0 size=0) to run queue because: pid=8, ppid=7, state=SUCCESS; AssignProcedure table=test, region=30eb0f9efacc61d337395de976dcb330, target=kalashnikov,54919,1544681197805 released the shared lock 2018-12-12 22:06:43,820 DEBUG [PEWorker-4] procedure.MasterProcedureScheduler(356): Add TableQueue(test, xlock=false sharedLock=0 size=1) to run queue because: the exclusive lock is not held by anyone when adding pid=7, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=test 2018-12-12 22:06:43,820 INFO [PEWorker-4] procedure2.ProcedureExecutor(1897): Finished subprocedure pid=8, resume processing parent pid=7, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=test 2018-12-12 22:06:43,820 DEBUG [PEWorker-16] procedure.MasterProcedureScheduler(366): Remove TableQueue(test, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=7, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=test 2018-12-12 22:06:43,821 INFO [PEWorker-4] procedure2.ProcedureExecutor(1485): Finished pid=8, ppid=7, state=SUCCESS; AssignProcedure table=test, region=30eb0f9efacc61d337395de976dcb330, target=kalashnikov,54919,1544681197805 in 410msec, unfinishedSiblingCount=0 2018-12-12 22:06:43,821 DEBUG [PEWorker-16] procedure.MasterProcedureScheduler(366): Remove TableQueue(test, xlock=true (7) sharedLock=0 size=0) from run queue because: pid=7, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=test held the exclusive lock 2018-12-12 22:06:43,823 DEBUG [PEWorker-16] hbase.MetaTableAccessor(2153): Put {"totalColumns":1,"row":"test","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1544681203823}]},"ts":1544681203823} 2018-12-12 22:06:43,826 INFO [PEWorker-16] hbase.MetaTableAccessor(1673): Updated tableName=test, state=ENABLED in hbase:meta 2018-12-12 22:06:43,827 DEBUG [PEWorker-16] procedure2.RootProcedureState(153): Add procedure pid=7, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=test as the 7th rollback step 2018-12-12 22:06:43,829 DEBUG [PEWorker-16] procedure2.RootProcedureState(153): Add procedure pid=7, state=SUCCESS, locked=true; CreateTableProcedure table=test as the 8th rollback step 2018-12-12 22:06:43,832 DEBUG [PEWorker-16] procedure.MasterProcedureScheduler(356): Add TableQueue(test, xlock=false sharedLock=0 size=0) to run queue because: pid=7, state=SUCCESS; CreateTableProcedure table=test released the exclusive lock 2018-12-12 22:06:43,832 INFO [PEWorker-16] procedure2.ProcedureExecutor(1485): Finished pid=7, state=SUCCESS; CreateTableProcedure table=test in 666msec 2018-12-12 22:06:43,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54915] master.MasterRpcServices(1179): Checking to see if procedure is done pid=7 2018-12-12 22:06:43,958 INFO [Time-limited test] client.HBaseAdmin$TableFuture(3666): Operation: CREATE, Table Name: default:test, procId: 7 completed 2018-12-12 22:06:43,958 DEBUG [Time-limited test] hbase.HBaseTestingUtility(3334): Waiting until all regions of table test get assigned. Timeout = 60000ms 2018-12-12 22:06:43,968 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2018-12-12 22:06:43,976 INFO [Time-limited test] hbase.HBaseTestingUtility(3386): All regions for table test assigned to meta. Checking AM states. 2018-12-12 22:06:43,977 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2018-12-12 22:06:43,977 INFO [Time-limited test] hbase.HBaseTestingUtility(3406): All regions for table test assigned. 2018-12-12 22:06:43,986 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=54915] master.HMaster(1898): Passed destination servername is null/empty so choosing a server at random 2018-12-12 22:06:43,987 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=54915] master.ServerManager(728): New admin connection to kalashnikov,54917,1544681197760 2018-12-12 22:06:43,996 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(556): Connection from 192.168.1.137:54959, version=2.1.2, sasl=false, ugi=stack (auth:SIMPLE), service=AdminService 2018-12-12 22:06:43,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54917] regionserver.RSRpcServices(2091): Warming up Region hbase:meta,,1.1588230740 2018-12-12 22:06:43,997 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54917] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-12-12 22:06:44,000 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54917] wal.AbstractFSWAL(414): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=kalashnikov%2C54917%2C1544681197760.meta, suffix=.meta, logDir=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/WALs/kalashnikov,54917,1544681197760, archiveDir=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/oldWALs 2018-12-12 22:06:44,009 DEBUG [RS-EventLoopGroup-4-5] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(783): SASL client skipping handshake in unsecured configuration for addr = /127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:54902,DS-b8b11ce4-1484-4801-8b0b-e3a38c78a69a,DISK] 2018-12-12 22:06:44,010 DEBUG [RS-EventLoopGroup-4-11] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(783): SASL client skipping handshake in unsecured configuration for addr = /127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:54906,DS-572a8465-16cf-4b8a-9cb7-4866d7586234,DISK] 2018-12-12 22:06:44,018 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54917] wal.AbstractFSWAL(672): New WAL /user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/WALs/kalashnikov,54917,1544681197760/kalashnikov%2C54917%2C1544681197760.meta.1544681204002.meta 2018-12-12 22:06:44,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54917] wal.AbstractFSWAL(762): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:54902,DS-b8b11ce4-1484-4801-8b0b-e3a38c78a69a,DISK], DatanodeInfoWithStorage[127.0.0.1:54906,DS-572a8465-16cf-4b8a-9cb7-4866d7586234,DISK]] 2018-12-12 22:06:44,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54917] regionserver.HRegion(7266): HRegion.Warming up region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2018-12-12 22:06:44,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54917] regionserver.HRegion(833): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-12-12 22:06:44,021 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/info 2018-12-12 22:06:44,021 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/info 2018-12-12 22:06:44,022 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(237): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-12-12 22:06:44,022 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-12-12 22:06:44,024 INFO [StoreOpener-1588230740-1] regionserver.HStore(332): Store=info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2018-12-12 22:06:44,025 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/rep_barrier 2018-12-12 22:06:44,025 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/rep_barrier 2018-12-12 22:06:44,026 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(237): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-12-12 22:06:44,027 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-12-12 22:06:44,028 INFO [StoreOpener-1588230740-1] regionserver.HStore(332): Store=rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2018-12-12 22:06:44,029 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/table 2018-12-12 22:06:44,029 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/table 2018-12-12 22:06:44,030 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(237): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-12-12 22:06:44,030 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-12-12 22:06:44,031 INFO [StoreOpener-1588230740-1] regionserver.HStore(332): Store=table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2018-12-12 22:06:44,036 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=54915] master.HMaster(1946): Client=stack//192.168.1.137 move hri=1588230740, source=kalashnikov,54919,1544681197805, destination=kalashnikov,54917,1544681197760, running balancer 2018-12-12 22:06:44,158 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=54915] procedure2.ProcedureExecutor(1092): Stored pid=9, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=1588230740, source=kalashnikov,54919,1544681197805, destination=kalashnikov,54917,1544681197760 2018-12-12 22:06:44,158 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=54915] procedure.MasterProcedureScheduler(356): Add MetaQueue(hbase:meta, xlock=false sharedLock=0 size=1) to run queue because: the exclusive lock is not held by anyone when adding pid=9, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=1588230740, source=kalashnikov,54919,1544681197805, destination=kalashnikov,54917,1544681197760 2018-12-12 22:06:44,159 DEBUG [PEWorker-6] procedure.MasterProcedureScheduler(366): Remove MetaQueue(hbase:meta, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=9, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=1588230740, source=kalashnikov,54919,1544681197805, destination=kalashnikov,54917,1544681197760 2018-12-12 22:06:44,159 INFO [PEWorker-6] procedure.MasterProcedureScheduler(741): Took xlock for pid=9, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure hri=1588230740, source=kalashnikov,54919,1544681197805, destination=kalashnikov,54917,1544681197760 2018-12-12 22:06:44,162 INFO [PEWorker-6] procedure2.ProcedureExecutor(1758): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=hbase:meta, region=1588230740, override=true, server=kalashnikov,54919,1544681197805}] 2018-12-12 22:06:44,162 DEBUG [PEWorker-6] procedure2.RootProcedureState(153): Add procedure pid=9, state=WAITING:MOVE_REGION_ASSIGN, locked=true; MoveRegionProcedure hri=1588230740, source=kalashnikov,54919,1544681197805, destination=kalashnikov,54917,1544681197760 as the 0th rollback step 2018-12-12 22:06:44,171 DEBUG [PEWorker-6] procedure.MasterProcedureScheduler(356): Add MetaQueue(hbase:meta, xlock=false sharedLock=0 size=1) to run queue because: the exclusive lock is not held by anyone when adding pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=hbase:meta, region=1588230740, override=true, server=kalashnikov,54919,1544681197805 2018-12-12 22:06:44,171 DEBUG [PEWorker-8] procedure.MasterProcedureScheduler(366): Remove MetaQueue(hbase:meta, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=hbase:meta, region=1588230740, override=true, server=kalashnikov,54919,1544681197805 2018-12-12 22:06:44,172 INFO [PEWorker-8] procedure.MasterProcedureScheduler(741): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=hbase:meta, region=1588230740, override=true, server=kalashnikov,54919,1544681197805 2018-12-12 22:06:44,174 INFO [PEWorker-8] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as kalashnikov,54919,1544681197805 2018-12-12 22:06:44,175 INFO [PEWorker-8] assignment.RegionTransitionProcedure(267): Dispatch pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; UnassignProcedure table=hbase:meta, region=1588230740, override=true, server=kalashnikov,54919,1544681197805 2018-12-12 22:06:44,175 DEBUG [PEWorker-8] procedure2.RootProcedureState(153): Add procedure pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; UnassignProcedure table=hbase:meta, region=1588230740, override=true, server=kalashnikov,54919,1544681197805 as the 1th rollback step 2018-12-12 22:06:44,214 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=54915] procedure.ProcedureSyncWait(198): waitFor pid=9 2018-12-12 22:06:44,342 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=54919] regionserver.RSRpcServices(1611): Close 1588230740, moving to kalashnikov,54917,1544681197760 2018-12-12 22:06:44,344 DEBUG [RS_CLOSE_META-regionserver/kalashnikov:0-0] regionserver.HRegion(1541): Closing 1588230740, disabling compactions & flushes 2018-12-12 22:06:44,344 DEBUG [RS_CLOSE_META-regionserver/kalashnikov:0-0] regionserver.HRegion(1581): Updates disabled for region hbase:meta,,1.1588230740 2018-12-12 22:06:44,345 INFO [RS_CLOSE_META-regionserver/kalashnikov:0-0] regionserver.HRegion(2617): Flushing 3/3 column families, dataSize=2.33 KB heapSize=4.70 KB 2018-12-12 22:06:44,423 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54902 is added to blk_1073741838_1014{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-062f61d8-8f41-4a9e-a2ce-6913541fccba:NORMAL:127.0.0.1:54906|RBW], ReplicaUC[[DISK]DS-dfa0bd22-3ff7-4df8-ab3c-2883ad1c4339:NORMAL:127.0.0.1:54902|FINALIZED]]} size 0 2018-12-12 22:06:44,425 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54906 is added to blk_1073741838_1014 size 6981 2018-12-12 22:06:44,425 INFO [RS_CLOSE_META-regionserver/kalashnikov:0-0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=2.17 KB at sequenceid=14 (bloomFilter=false), to=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/.tmp/info/838d862548964b0f8587a2ea4b4e3947 2018-12-12 22:06:44,498 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54906 is added to blk_1073741839_1015{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b8b11ce4-1484-4801-8b0b-e3a38c78a69a:NORMAL:127.0.0.1:54902|RBW], ReplicaUC[[DISK]DS-062f61d8-8f41-4a9e-a2ce-6913541fccba:NORMAL:127.0.0.1:54906|RBW]]} size 0 2018-12-12 22:06:44,498 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54902 is added to blk_1073741839_1015{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b8b11ce4-1484-4801-8b0b-e3a38c78a69a:NORMAL:127.0.0.1:54902|RBW], ReplicaUC[[DISK]DS-062f61d8-8f41-4a9e-a2ce-6913541fccba:NORMAL:127.0.0.1:54906|RBW]]} size 0 2018-12-12 22:06:44,499 INFO [RS_CLOSE_META-regionserver/kalashnikov:0-0] regionserver.DefaultStoreFlusher(85): Flushed memstore data size=166 B at sequenceid=14 (bloomFilter=false), to=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/.tmp/table/f304450145df410295cac33ae1d05ecc 2018-12-12 22:06:44,509 DEBUG [RS_CLOSE_META-regionserver/kalashnikov:0-0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/.tmp/info/838d862548964b0f8587a2ea4b4e3947 as hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/info/838d862548964b0f8587a2ea4b4e3947 2018-12-12 22:06:44,517 INFO [RS_CLOSE_META-regionserver/kalashnikov:0-0] regionserver.HStore(1074): Added hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/info/838d862548964b0f8587a2ea4b4e3947, entries=20, sequenceid=14, filesize=6.8 K 2018-12-12 22:06:44,524 DEBUG [RS_CLOSE_META-regionserver/kalashnikov:0-0] regionserver.HRegionFileSystem(464): Committing hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/.tmp/table/f304450145df410295cac33ae1d05ecc as hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/table/f304450145df410295cac33ae1d05ecc 2018-12-12 22:06:44,533 INFO [RS_CLOSE_META-regionserver/kalashnikov:0-0] regionserver.HStore(1074): Added hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/table/f304450145df410295cac33ae1d05ecc, entries=4, sequenceid=14, filesize=4.7 K 2018-12-12 22:06:44,535 INFO [RS_CLOSE_META-regionserver/kalashnikov:0-0] regionserver.HRegion(2816): Finished flush of dataSize ~2.33 KB/2383, heapSize ~4.52 KB/4624, currentSize=0 B/0 for 1588230740 in 190ms, sequenceid=14, compaction requested=false 2018-12-12 22:06:44,545 DEBUG [RS_CLOSE_META-regionserver/kalashnikov:0-0] wal.WALSplitter(695): Wrote file=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/hbase/meta/1588230740/recovered.edits/17.seqid, newMaxSeqId=17, maxSeqId=1 2018-12-12 22:06:44,546 DEBUG [RS_CLOSE_META-regionserver/kalashnikov:0-0] coprocessor.CoprocessorHost(288): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-12-12 22:06:44,547 INFO [RS_CLOSE_META-regionserver/kalashnikov:0-0] regionserver.HRegion(1698): Closed hbase:meta,,1.1588230740 2018-12-12 22:06:44,548 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=54915] assignment.RegionTransitionProcedure(290): Received report CLOSED seqId=-1, pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; UnassignProcedure table=hbase:meta, region=1588230740, override=true, server=kalashnikov,54919,1544681197805; rit=CLOSING, location=kalashnikov,54919,1544681197805 2018-12-12 22:06:44,549 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=54915] procedure.MasterProcedureScheduler(356): Add MetaQueue(hbase:meta, xlock=false sharedLock=0 size=1) to run queue because: pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_FINISH, locked=true; UnassignProcedure table=hbase:meta, region=1588230740, override=true, server=kalashnikov,54919,1544681197805 has lock 2018-12-12 22:06:44,549 DEBUG [PEWorker-10] procedure.MasterProcedureScheduler(366): Remove MetaQueue(hbase:meta, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_FINISH, locked=true; UnassignProcedure table=hbase:meta, region=1588230740, override=true, server=kalashnikov,54919,1544681197805 2018-12-12 22:06:44,549 DEBUG [PEWorker-10] assignment.RegionTransitionProcedure(387): Finishing pid=10, ppid=9, state=RUNNABLE:REGION_TRANSITION_FINISH, locked=true; UnassignProcedure table=hbase:meta, region=1588230740, override=true, server=kalashnikov,54919,1544681197805; rit=CLOSING, location=kalashnikov,54919,1544681197805 2018-12-12 22:06:44,549 WARN [PEWorker-10] zookeeper.MetaTableLocator(449): Tried to set null ServerName in hbase:meta; skipping -- ServerName required 2018-12-12 22:06:44,549 DEBUG [PEWorker-10] procedure2.RootProcedureState(153): Add procedure pid=10, ppid=9, state=SUCCESS, locked=true; UnassignProcedure table=hbase:meta, region=1588230740, override=true, server=kalashnikov,54919,1544681197805 as the 2th rollback step 2018-12-12 22:06:44,550 DEBUG [RS_CLOSE_META-regionserver/kalashnikov:0-0] handler.CloseRegionHandler(124): Closed hbase:meta,,1.1588230740 2018-12-12 22:06:44,553 DEBUG [PEWorker-10] procedure.MasterProcedureScheduler(356): Add MetaQueue(hbase:meta, xlock=false sharedLock=0 size=1) to run queue because: pid=9, state=RUNNABLE:MOVE_REGION_ASSIGN, locked=true; MoveRegionProcedure hri=1588230740, source=kalashnikov,54919,1544681197805, destination=kalashnikov,54917,1544681197760 has lock 2018-12-12 22:06:44,553 INFO [PEWorker-10] procedure2.ProcedureExecutor(1897): Finished subprocedure pid=10, resume processing parent pid=9, state=RUNNABLE:MOVE_REGION_ASSIGN, locked=true; MoveRegionProcedure hri=1588230740, source=kalashnikov,54919,1544681197805, destination=kalashnikov,54917,1544681197760 2018-12-12 22:06:44,553 DEBUG [PEWorker-11] procedure.MasterProcedureScheduler(366): Remove MetaQueue(hbase:meta, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=9, state=RUNNABLE:MOVE_REGION_ASSIGN, locked=true; MoveRegionProcedure hri=1588230740, source=kalashnikov,54919,1544681197805, destination=kalashnikov,54917,1544681197760 2018-12-12 22:06:44,553 INFO [PEWorker-10] procedure2.ProcedureExecutor(1485): Finished pid=10, ppid=9, state=SUCCESS; UnassignProcedure table=hbase:meta, region=1588230740, override=true, server=kalashnikov,54919,1544681197805 in 387msec, unfinishedSiblingCount=0 2018-12-12 22:06:44,554 INFO [PEWorker-11] procedure2.ProcedureExecutor(1758): Initialized subprocedures=[{pid=11, ppid=9, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740, target=kalashnikov,54917,1544681197760}] 2018-12-12 22:06:44,554 DEBUG [PEWorker-11] procedure2.RootProcedureState(153): Add procedure pid=9, state=WAITING, locked=true; MoveRegionProcedure hri=1588230740, source=kalashnikov,54919,1544681197805, destination=kalashnikov,54917,1544681197760 as the 3th rollback step 2018-12-12 22:06:44,555 DEBUG [PEWorker-11] procedure.MasterProcedureScheduler(356): Add MetaQueue(hbase:meta, xlock=false sharedLock=0 size=1) to run queue because: the exclusive lock is not held by anyone when adding pid=11, ppid=9, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740, target=kalashnikov,54917,1544681197760 2018-12-12 22:06:44,555 DEBUG [PEWorker-11] procedure.MasterProcedureScheduler(366): Remove MetaQueue(hbase:meta, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=11, ppid=9, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740, target=kalashnikov,54917,1544681197760 2018-12-12 22:06:44,556 INFO [PEWorker-11] procedure.MasterProcedureScheduler(741): Took xlock for pid=11, ppid=9, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740, target=kalashnikov,54917,1544681197760 2018-12-12 22:06:44,557 INFO [PEWorker-11] assignment.AssignProcedure(254): Starting pid=11, ppid=9, state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure table=hbase:meta, region=1588230740, target=kalashnikov,54917,1544681197760; rit=OFFLINE, location=kalashnikov,54917,1544681197760; forceNewPlan=false, retain=false 2018-12-12 22:06:44,557 DEBUG [PEWorker-11] procedure2.RootProcedureState(153): Add procedure pid=11, ppid=9, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:meta, region=1588230740, target=kalashnikov,54917,1544681197760 as the 4th rollback step 2018-12-12 22:06:44,711 INFO [master/kalashnikov:0] balancer.BaseLoadBalancer(1531): Reassigned 1 regions. 1 retained the pre-restart assignment. 2018-12-12 22:06:44,711 DEBUG [master/kalashnikov:0] procedure.MasterProcedureScheduler(356): Add MetaQueue(hbase:meta, xlock=false sharedLock=0 size=1) to run queue because: pid=11, ppid=9, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:meta, region=1588230740, target=kalashnikov,54917,1544681197760 has lock 2018-12-12 22:06:44,711 DEBUG [PEWorker-10] procedure.MasterProcedureScheduler(366): Remove MetaQueue(hbase:meta, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=11, ppid=9, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:meta, region=1588230740, target=kalashnikov,54917,1544681197760 2018-12-12 22:06:44,713 INFO [PEWorker-10] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as kalashnikov,54917,1544681197760 2018-12-12 22:06:44,720 INFO [PEWorker-10] assignment.RegionTransitionProcedure(267): Dispatch pid=11, ppid=9, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:meta, region=1588230740, target=kalashnikov,54917,1544681197760 2018-12-12 22:06:44,720 DEBUG [PEWorker-10] procedure2.RootProcedureState(153): Add procedure pid=11, ppid=9, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:meta, region=1588230740, target=kalashnikov,54917,1544681197760 as the 5th rollback step 2018-12-12 22:06:44,875 INFO [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=54917] regionserver.RSRpcServices(1987): Open hbase:meta,,1.1588230740 2018-12-12 22:06:44,876 DEBUG [RS_OPEN_META-regionserver/kalashnikov:0-0] regionserver.HRegion(7177): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2018-12-12 22:06:44,877 DEBUG [RS_OPEN_META-regionserver/kalashnikov:0-0] coprocessor.CoprocessorHost(200): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2018-12-12 22:06:44,877 DEBUG [RS_OPEN_META-regionserver/kalashnikov:0-0] regionserver.HRegion(8154): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2018-12-12 22:06:44,878 INFO [RS_OPEN_META-regionserver/kalashnikov:0-0] regionserver.RegionCoprocessorHost(394): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2018-12-12 22:06:44,878 DEBUG [RS_OPEN_META-regionserver/kalashnikov:0-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table meta 1588230740 2018-12-12 22:06:44,878 DEBUG [RS_OPEN_META-regionserver/kalashnikov:0-0] regionserver.HRegion(833): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2018-12-12 22:06:44,882 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/info 2018-12-12 22:06:44,882 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/info 2018-12-12 22:06:44,883 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(237): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-12-12 22:06:44,884 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-12-12 22:06:44,895 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(584): loaded hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/info/838d862548964b0f8587a2ea4b4e3947 2018-12-12 22:06:44,896 INFO [StoreOpener-1588230740-1] regionserver.HStore(332): Store=info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2018-12-12 22:06:44,897 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/rep_barrier 2018-12-12 22:06:44,897 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/rep_barrier 2018-12-12 22:06:44,898 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(237): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-12-12 22:06:44,898 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-12-12 22:06:44,899 INFO [StoreOpener-1588230740-1] regionserver.HStore(332): Store=rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2018-12-12 22:06:44,900 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/table 2018-12-12 22:06:44,900 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(598): Set storagePolicy=HOT for path=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/table 2018-12-12 22:06:44,901 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(237): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=747.70 KB, freeSize=994.87 MB, maxSize=995.60 MB, heapSize=747.70 KB, minSize=945.82 MB, minFactor=0.95, multiSize=472.91 MB, multiFactor=0.5, singleSize=236.46 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-12-12 22:06:44,901 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(147): size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-12-12 22:06:44,910 DEBUG [StoreOpener-1588230740-1] regionserver.HStore(584): loaded hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740/table/f304450145df410295cac33ae1d05ecc 2018-12-12 22:06:44,910 INFO [StoreOpener-1588230740-1] regionserver.HStore(332): Store=table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2018-12-12 22:06:44,912 DEBUG [RS_OPEN_META-regionserver/kalashnikov:0-0] regionserver.HRegion(4571): Found 0 recovered edits file(s) under hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/data/hbase/meta/1588230740 2018-12-12 22:06:44,914 DEBUG [RS_OPEN_META-regionserver/kalashnikov:0-0] regionserver.HRegion(4571): Found 0 recovered edits file(s) under hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/hbase/meta/1588230740 2018-12-12 22:06:44,916 DEBUG [RS_OPEN_META-regionserver/kalashnikov:0-0] regionserver.FlushLargeStoresPolicy(61): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7M)) instead. 2018-12-12 22:06:44,917 DEBUG [RS_OPEN_META-regionserver/kalashnikov:0-0] regionserver.HRegion(998): writing seq id for 1588230740 2018-12-12 22:06:44,918 INFO [RS_OPEN_META-regionserver/kalashnikov:0-0] regionserver.HRegion(1002): Opened 1588230740; next sequenceid=18 2018-12-12 22:06:44,920 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2177): Post open deploy tasks for hbase:meta,,1.1588230740 2018-12-12 22:06:44,922 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=54915] assignment.RegionTransitionProcedure(290): Received report OPENED seqId=18, pid=11, ppid=9, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:meta, region=1588230740, target=kalashnikov,54917,1544681197760; rit=OPENING, location=kalashnikov,54917,1544681197760 2018-12-12 22:06:44,922 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=4,queue=0,port=54915] procedure.MasterProcedureScheduler(356): Add MetaQueue(hbase:meta, xlock=false sharedLock=0 size=1) to run queue because: pid=11, ppid=9, state=RUNNABLE:REGION_TRANSITION_FINISH, locked=true; AssignProcedure table=hbase:meta, region=1588230740, target=kalashnikov,54917,1544681197760 has lock 2018-12-12 22:06:44,923 DEBUG [PEWorker-15] procedure.MasterProcedureScheduler(366): Remove MetaQueue(hbase:meta, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=11, ppid=9, state=RUNNABLE:REGION_TRANSITION_FINISH, locked=true; AssignProcedure table=hbase:meta, region=1588230740, target=kalashnikov,54917,1544681197760 2018-12-12 22:06:44,923 DEBUG [PEWorker-15] assignment.RegionTransitionProcedure(387): Finishing pid=11, ppid=9, state=RUNNABLE:REGION_TRANSITION_FINISH, locked=true; AssignProcedure table=hbase:meta, region=1588230740, target=kalashnikov,54917,1544681197760; rit=OPENING, location=kalashnikov,54917,1544681197760 2018-12-12 22:06:44,923 INFO [PEWorker-15] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as kalashnikov,54917,1544681197760 2018-12-12 22:06:44,923 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2201): Finished post open deploy task for hbase:meta,,1.1588230740 2018-12-12 22:06:44,924 DEBUG [RS_OPEN_META-regionserver/kalashnikov:0-0] handler.OpenRegionHandler(127): Opened hbase:meta,,1.1588230740 on kalashnikov,54917,1544681197760 2018-12-12 22:06:44,931 DEBUG [PEWorker-15] procedure2.RootProcedureState(153): Add procedure pid=11, ppid=9, state=SUCCESS, locked=true; AssignProcedure table=hbase:meta, region=1588230740, target=kalashnikov,54917,1544681197760 as the 6th rollback step 2018-12-12 22:06:44,935 DEBUG [PEWorker-15] procedure.MasterProcedureScheduler(356): Add MetaQueue(hbase:meta, xlock=false sharedLock=0 size=1) to run queue because: pid=9, state=RUNNABLE, locked=true; MoveRegionProcedure hri=1588230740, source=kalashnikov,54919,1544681197805, destination=kalashnikov,54917,1544681197760 has lock 2018-12-12 22:06:44,935 INFO [PEWorker-15] procedure2.ProcedureExecutor(1897): Finished subprocedure pid=11, resume processing parent pid=9, state=RUNNABLE, locked=true; MoveRegionProcedure hri=1588230740, source=kalashnikov,54919,1544681197805, destination=kalashnikov,54917,1544681197760 2018-12-12 22:06:44,935 DEBUG [PEWorker-1] procedure.MasterProcedureScheduler(366): Remove MetaQueue(hbase:meta, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=9, state=RUNNABLE, locked=true; MoveRegionProcedure hri=1588230740, source=kalashnikov,54919,1544681197805, destination=kalashnikov,54917,1544681197760 2018-12-12 22:06:44,935 INFO [PEWorker-15] procedure2.ProcedureExecutor(1485): Finished pid=11, ppid=9, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740, target=kalashnikov,54917,1544681197760 in 377msec, unfinishedSiblingCount=0 2018-12-12 22:06:44,935 DEBUG [PEWorker-1] procedure2.RootProcedureState(153): Add procedure pid=9, state=SUCCESS, locked=true; MoveRegionProcedure hri=1588230740, source=kalashnikov,54919,1544681197805, destination=kalashnikov,54917,1544681197760 as the 7th rollback step 2018-12-12 22:06:44,938 DEBUG [PEWorker-1] procedure.MasterProcedureScheduler(356): Add TableQueue(hbase:meta, xlock=false sharedLock=0 size=0) to run queue because: pid=9, state=SUCCESS; MoveRegionProcedure hri=1588230740, source=kalashnikov,54919,1544681197805, destination=kalashnikov,54917,1544681197760 released the shared lock 2018-12-12 22:06:44,938 INFO [PEWorker-1] procedure2.ProcedureExecutor(1485): Finished pid=9, state=SUCCESS; MoveRegionProcedure hri=1588230740, source=kalashnikov,54919,1544681197805, destination=kalashnikov,54917,1544681197760 in 898msec 2018-12-12 22:06:44,954 INFO [Time-limited test] hbase.MiniHBaseCluster(278): Killing kalashnikov,54919,1544681197805 2018-12-12 22:06:44,958 ERROR [Time-limited test] helpers.MarkerIgnoringBase(143): ***** ABORTING region server kalashnikov,54919,1544681197805: Simulated kill ***** 2018-12-12 22:06:44,958 ERROR [Time-limited test] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-12-12 22:06:44,965 INFO [Time-limited test] regionserver.HRegionServer(2361): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 684720128, "init" : 268435456, "max" : 2609905664, "used" : 428419744 }, "NonHeapMemoryUsage" : { "committed" : 106446848, "init" : 2555904, "max" : -1, "used" : 104679400 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "kalashnikov" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "kalashnikov" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.Context" : "regionserver", "tag.Hostname" : "kalashnikov" } ] } 2018-12-12 22:06:44,969 WARN [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=54915] master.MasterRpcServices(541): kalashnikov,54919,1544681197805 reported a fatal error: ***** ABORTING region server kalashnikov,54919,1544681197805: Simulated kill ***** 2018-12-12 22:06:44,973 INFO [Time-limited test] regionserver.HRegionServer(2138): ***** STOPPING region server 'kalashnikov,54919,1544681197805' ***** 2018-12-12 22:06:44,974 INFO [Time-limited test] regionserver.HRegionServer(2152): STOPPED: Simulated kill 2018-12-12 22:06:44,974 INFO [RS:1;kalashnikov:54919] regionserver.SplitLogWorker(166): Sending interrupt to stop the worker thread 2018-12-12 22:06:44,974 INFO [SplitLogWorker-kalashnikov:54919] regionserver.SplitLogWorker(148): SplitLogWorker interrupted. Exiting. 2018-12-12 22:06:44,974 INFO [RS:1;kalashnikov:54919] regionserver.HeapMemoryManager(221): Stopping 2018-12-12 22:06:44,974 INFO [SplitLogWorker-kalashnikov:54919] regionserver.SplitLogWorker(157): SplitLogWorker kalashnikov,54919,1544681197805 exiting 2018-12-12 22:06:44,975 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(382): MemStoreFlusher.0 exiting 2018-12-12 22:06:44,975 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(382): MemStoreFlusher.1 exiting 2018-12-12 22:06:44,975 INFO [RS:1;kalashnikov:54919] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager abruptly. 2018-12-12 22:06:44,975 INFO [Time-limited test] hbase.Waiter(189): Waiting up to [10,000] milli-secs(wait.for.ratio=[1]) 2018-12-12 22:06:44,975 INFO [RS:1;kalashnikov:54919] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager abruptly. 2018-12-12 22:06:44,975 DEBUG [RS:1;kalashnikov:54919] zookeeper.MetaTableLocator(642): Stopping MetaTableLocator 2018-12-12 22:06:44,975 INFO [RS:1;kalashnikov:54919] zookeeper.ReadOnlyZKClient(350): Close zookeeper connection 0x45ed407e to localhost:63470 2018-12-12 22:06:44,975 DEBUG [RS:1;kalashnikov:54919] ipc.AbstractRpcClient(483): Stopping rpc client 2018-12-12 22:06:44,983 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54906 is added to blk_1073741832_1008{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dfa0bd22-3ff7-4df8-ab3c-2883ad1c4339:NORMAL:127.0.0.1:54902|RBW], ReplicaUC[[DISK]DS-572a8465-16cf-4b8a-9cb7-4866d7586234:NORMAL:127.0.0.1:54906|RBW]]} size 0 2018-12-12 22:06:44,983 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54902 is added to blk_1073741832_1008{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dfa0bd22-3ff7-4df8-ab3c-2883ad1c4339:NORMAL:127.0.0.1:54902|RBW], ReplicaUC[[DISK]DS-572a8465-16cf-4b8a-9cb7-4866d7586234:NORMAL:127.0.0.1:54906|RBW]]} size 0 2018-12-12 22:06:44,992 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54902 is added to blk_1073741831_1007{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-572a8465-16cf-4b8a-9cb7-4866d7586234:NORMAL:127.0.0.1:54906|RBW], ReplicaUC[[DISK]DS-dfa0bd22-3ff7-4df8-ab3c-2883ad1c4339:NORMAL:127.0.0.1:54902|RBW]]} size 0 2018-12-12 22:06:44,992 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54906 is added to blk_1073741831_1007{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-572a8465-16cf-4b8a-9cb7-4866d7586234:NORMAL:127.0.0.1:54906|RBW], ReplicaUC[[DISK]DS-dfa0bd22-3ff7-4df8-ab3c-2883ad1c4339:NORMAL:127.0.0.1:54902|RBW]]} size 0 2018-12-12 22:06:44,993 DEBUG [RS:1;kalashnikov:54919] ipc.AbstractRpcClient(483): Stopping rpc client 2018-12-12 22:06:44,994 INFO [RS:1;kalashnikov:54919] regionserver.Leases(149): Closed leases 2018-12-12 22:06:44,994 INFO [RS:1;kalashnikov:54919] ipc.NettyRpcServer(144): Stopping server on /192.168.1.137:54919 2018-12-12 22:06:45,001 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:54915-0x167a62c89310000, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-12-12 22:06:45,001 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:54917-0x167a62c89310001, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/kalashnikov,54919,1544681197805 2018-12-12 22:06:45,001 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:54919-0x167a62c89310002, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/kalashnikov,54919,1544681197805 2018-12-12 22:06:45,001 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:54919-0x167a62c89310002, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-12-12 22:06:45,002 INFO [RS:1;kalashnikov:54919] regionserver.HRegionServer(1154): Exiting; stopping=kalashnikov,54919,1544681197805; zookeeper connection closed. 2018-12-12 22:06:45,002 INFO [RegionServerTracker-0] master.RegionServerTracker(171): RegionServer ephemeral node deleted, processing expiration [kalashnikov,54919,1544681197805] 2018-12-12 22:06:45,002 DEBUG [RegionServerTracker-0] master.DeadServer(143): Added kalashnikov,54919,1544681197805; numProcessing=1 2018-12-12 22:06:45,002 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@7ef9ac47] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(221): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@7ef9ac47 2018-12-12 22:06:45,002 INFO [RegionServerTracker-0] master.ServerManager(586): Processing expiration of kalashnikov,54919,1544681197805 on kalashnikov,54915,1544681196595 2018-12-12 22:06:45,002 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:54917-0x167a62c89310001, quorum=localhost:63470, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/kalashnikov,54917,1544681197760 2018-12-12 22:06:45,003 INFO [Time-limited test-EventThread] replication.ReplicationTrackerZKImpl$OtherRegionServerWatcher(122): /hbase/rs/kalashnikov,54919,1544681197805 znode expired, triggering replicatorRemoved event 2018-12-12 22:06:45,005 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:54917-0x167a62c89310001, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-12-12 22:06:45,006 DEBUG [Time-limited test-EventThread] zookeeper.ZKUtil(355): regionserver:54917-0x167a62c89310001, quorum=localhost:63470, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/kalashnikov,54917,1544681197760 2018-12-12 22:06:45,125 DEBUG [RegionServerTracker-0] procedure2.ProcedureExecutor(1092): Stored pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure server=kalashnikov,54919,1544681197805, splitWal=true, meta=false 2018-12-12 22:06:45,125 DEBUG [RegionServerTracker-0] procedure.MasterProcedureScheduler(356): Add ServerQueue(kalashnikov,54919,1544681197805, xlock=false sharedLock=0 size=1) to run queue because: the exclusive lock is not held by anyone when adding pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure server=kalashnikov,54919,1544681197805, splitWal=true, meta=false 2018-12-12 22:06:45,125 DEBUG [RegionServerTracker-0] assignment.AssignmentManager(1337): Added=kalashnikov,54919,1544681197805 to dead servers, submitted shutdown handler to be executed meta=false 2018-12-12 22:06:45,127 DEBUG [PEWorker-15] procedure.MasterProcedureScheduler(366): Remove ServerQueue(kalashnikov,54919,1544681197805, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure server=kalashnikov,54919,1544681197805, splitWal=true, meta=false 2018-12-12 22:06:45,128 DEBUG [PEWorker-15] procedure.MasterProcedureScheduler(366): Remove ServerQueue(kalashnikov,54919,1544681197805, xlock=true (12) sharedLock=0 size=0) from run queue because: pid=12, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure server=kalashnikov,54919,1544681197805, splitWal=true, meta=false held exclusive lock 2018-12-12 22:06:45,130 DEBUG [PEWorker-15] master.DeadServer(157): Started processing kalashnikov,54919,1544681197805; numProcessing=1 2018-12-12 22:06:45,131 INFO [PEWorker-15] procedure.ServerCrashProcedure(119): Start pid=12, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=kalashnikov,54919,1544681197805, splitWal=true, meta=false 2018-12-12 22:06:45,131 DEBUG [PEWorker-15] procedure2.RootProcedureState(153): Add procedure pid=12, state=RUNNABLE:SERVER_CRASH_GET_REGIONS, locked=true; ServerCrashProcedure server=kalashnikov,54919,1544681197805, splitWal=true, meta=false as the 0th rollback step 2018-12-12 22:06:45,134 DEBUG [PEWorker-15] procedure.MasterProcedureScheduler(356): Add ServerQueue(kalashnikov,54919,1544681197805, xlock=false sharedLock=0 size=0) to run queue because: pid=12, state=RUNNABLE:SERVER_CRASH_GET_REGIONS; ServerCrashProcedure server=kalashnikov,54919,1544681197805, splitWal=true, meta=false released exclusive lock 2018-12-12 22:06:45,134 DEBUG [PEWorker-15] procedure.MasterProcedureScheduler(356): Add ServerQueue(kalashnikov,54919,1544681197805, xlock=false sharedLock=0 size=1) to run queue because: the exclusive lock is not held by anyone when adding pid=12, state=RUNNABLE:SERVER_CRASH_GET_REGIONS; ServerCrashProcedure server=kalashnikov,54919,1544681197805, splitWal=true, meta=false 2018-12-12 22:06:45,135 DEBUG [PEWorker-15] procedure.MasterProcedureScheduler(366): Remove ServerQueue(kalashnikov,54919,1544681197805, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=12, state=RUNNABLE:SERVER_CRASH_GET_REGIONS; ServerCrashProcedure server=kalashnikov,54919,1544681197805, splitWal=true, meta=false 2018-12-12 22:06:45,135 DEBUG [PEWorker-15] procedure.MasterProcedureScheduler(366): Remove ServerQueue(kalashnikov,54919,1544681197805, xlock=true (12) sharedLock=0 size=0) from run queue because: pid=12, state=RUNNABLE:SERVER_CRASH_GET_REGIONS; ServerCrashProcedure server=kalashnikov,54919,1544681197805, splitWal=true, meta=false held exclusive lock 2018-12-12 22:06:45,136 DEBUG [PEWorker-15] procedure2.RootProcedureState(153): Add procedure pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure server=kalashnikov,54919,1544681197805, splitWal=true, meta=false as the 1th rollback step 2018-12-12 22:06:45,138 DEBUG [PEWorker-15] procedure.MasterProcedureScheduler(356): Add ServerQueue(kalashnikov,54919,1544681197805, xlock=false sharedLock=0 size=0) to run queue because: pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS; ServerCrashProcedure server=kalashnikov,54919,1544681197805, splitWal=true, meta=false released exclusive lock 2018-12-12 22:06:45,138 DEBUG [PEWorker-15] procedure.MasterProcedureScheduler(356): Add ServerQueue(kalashnikov,54919,1544681197805, xlock=false sharedLock=0 size=1) to run queue because: the exclusive lock is not held by anyone when adding pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS; ServerCrashProcedure server=kalashnikov,54919,1544681197805, splitWal=true, meta=false 2018-12-12 22:06:45,138 DEBUG [PEWorker-9] procedure.MasterProcedureScheduler(366): Remove ServerQueue(kalashnikov,54919,1544681197805, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS; ServerCrashProcedure server=kalashnikov,54919,1544681197805, splitWal=true, meta=false 2018-12-12 22:06:45,139 DEBUG [PEWorker-9] procedure.MasterProcedureScheduler(366): Remove ServerQueue(kalashnikov,54919,1544681197805, xlock=true (12) sharedLock=0 size=0) from run queue because: pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS; ServerCrashProcedure server=kalashnikov,54919,1544681197805, splitWal=true, meta=false held exclusive lock 2018-12-12 22:06:45,140 DEBUG [PEWorker-9] procedure.ServerCrashProcedure(247): Splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure server=kalashnikov,54919,1544681197805, splitWal=true, meta=false 2018-12-12 22:06:45,143 DEBUG [PEWorker-9] master.MasterWalManager(365): Renamed region directory: hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/WALs/kalashnikov,54919,1544681197805-splitting 2018-12-12 22:06:45,143 DEBUG [PEWorker-9] master.SplitLogManager(465): Dead splitlog workers [kalashnikov,54919,1544681197805] 2018-12-12 22:06:45,145 INFO [PEWorker-9] master.SplitLogManager(245): Started splitting 1 logs in [hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/WALs/kalashnikov,54919,1544681197805-splitting] for [kalashnikov,54919,1544681197805] 2018-12-12 22:06:45,154 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:54917-0x167a62c89310001, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-12-12 22:06:45,155 DEBUG [SplitLogWorker-kalashnikov:54917] coordination.ZkSplitLogWorkerCoordination(436): Current region server kalashnikov,54917,1544681197760 is ready to take more tasks, will get task list and try grab tasks again. 2018-12-12 22:06:45,155 DEBUG [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination(332): Put up splitlog task at znode /hbase/splitWAL/WALs%2Fkalashnikov%2C54919%2C1544681197805-splitting%2Fkalashnikov%252C54919%252C1544681197805.1544681200975 2018-12-12 22:06:45,157 INFO [SplitLogWorker-kalashnikov:54917] coordination.ZkSplitLogWorkerCoordination(258): worker kalashnikov,54917,1544681197760 acquired task /hbase/splitWAL/WALs%2Fkalashnikov%2C54919%2C1544681197805-splitting%2Fkalashnikov%252C54919%252C1544681197805.1544681200975 2018-12-12 22:06:45,158 INFO [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination(454): Task /hbase/splitWAL/WALs%2Fkalashnikov%2C54919%2C1544681197805-splitting%2Fkalashnikov%252C54919%252C1544681197805.1544681200975 acquired by kalashnikov,54917,1544681197760 2018-12-12 22:06:45,196 INFO [master/kalashnikov:0.splitLogManager..Chore.1] master.SplitLogManager$TimeoutMonitor(595): total=1, unassigned=0, tasks={/hbase/splitWAL/WALs%2Fkalashnikov%2C54919%2C1544681197805-splitting%2Fkalashnikov%252C54919%252C1544681197805.1544681200975=last_update = 1544681205158 last_version = 1 cur_worker_name = kalashnikov,54917,1544681197760 status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0} 2018-12-12 22:06:45,208 INFO [RS_LOG_REPLAY_OPS-regionserver/kalashnikov:0-0] wal.WALSplitter(250): Splitting WAL=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/WALs/kalashnikov,54919,1544681197805-splitting/kalashnikov%2C54919%2C1544681197805.1544681200975, length=822 2018-12-12 22:06:45,209 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:54917-0x167a62c89310001, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fkalashnikov%2C54919%2C1544681197805-splitting%2Fkalashnikov%252C54919%252C1544681197805.1544681200975 2018-12-12 22:06:45,209 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:54915-0x167a62c89310000, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fkalashnikov%2C54919%2C1544681197805-splitting%2Fkalashnikov%252C54919%252C1544681197805.1544681200975 2018-12-12 22:06:45,209 INFO [RS_LOG_REPLAY_OPS-regionserver/kalashnikov:0-0] util.FSHDFSUtils(195): Recover lease on dfs file hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/WALs/kalashnikov,54919,1544681197805-splitting/kalashnikov%2C54919%2C1544681197805.1544681200975 2018-12-12 22:06:45,214 INFO [RS_LOG_REPLAY_OPS-regionserver/kalashnikov:0-0] util.FSHDFSUtils(284): Recovered lease, attempt=0 on file=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/WALs/kalashnikov,54919,1544681197805-splitting/kalashnikov%2C54919%2C1544681197805.1544681200975 after 5ms 2018-12-12 22:06:45,234 DEBUG [RS_LOG_REPLAY_OPS-regionserver/kalashnikov:0-0] wal.WALSplitter(283): DLS Last flushed sequenceid for b61e07ebe62dcc1570e2963289f44b29: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "info" sequence_id: 3 } 2018-12-12 22:06:45,238 DEBUG [RS_LOG_REPLAY_OPS-regionserver/kalashnikov:0-0] wal.WALSplitter(283): DLS Last flushed sequenceid for 30eb0f9efacc61d337395de976dcb330: last_flushed_sequence_id: 18446744073709551615 store_sequence_id { family_name: "cf" sequence_id: 18446744073709551615 } 2018-12-12 22:06:45,238 DEBUG [RS_LOG_REPLAY_OPS-regionserver/kalashnikov:0-0] wal.WALSplitter(335): Finishing writing output logs and closing down 2018-12-12 22:06:45,238 DEBUG [RS_LOG_REPLAY_OPS-regionserver/kalashnikov:0-0] wal.WALSplitter$OutputSink(1179): Waiting for split writer threads to finish 2018-12-12 22:06:45,244 DEBUG [RS_LOG_REPLAY_OPS-regionserver/kalashnikov:0-0-Writer-2] wal.WALSplitter$LogRecoveredEditsOutputSink(1524): Creating writer path=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/hbase/namespace/b61e07ebe62dcc1570e2963289f44b29/recovered.edits/0000000000000000004-kalashnikov%2C54919%2C1544681197805.1544681200975.temp 2018-12-12 22:06:45,249 INFO [RS_LOG_REPLAY_OPS-regionserver/kalashnikov:0-0] wal.WALSplitter$OutputSink(1203): 3 split writers finished; closing. 2018-12-12 22:06:45,261 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54902 is added to blk_1073741840_1016{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-062f61d8-8f41-4a9e-a2ce-6913541fccba:NORMAL:127.0.0.1:54906|RBW], ReplicaUC[[DISK]DS-dfa0bd22-3ff7-4df8-ab3c-2883ad1c4339:NORMAL:127.0.0.1:54902|FINALIZED]]} size 0 2018-12-12 22:06:45,262 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54906 is added to blk_1073741840_1016{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dfa0bd22-3ff7-4df8-ab3c-2883ad1c4339:NORMAL:127.0.0.1:54902|FINALIZED], ReplicaUC[[DISK]DS-572a8465-16cf-4b8a-9cb7-4866d7586234:NORMAL:127.0.0.1:54906|FINALIZED]]} size 0 2018-12-12 22:06:45,263 DEBUG [split-log-closeStream-1] wal.WALSplitter$LogRecoveredEditsOutputSink(1397): Closed wap hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/hbase/namespace/b61e07ebe62dcc1570e2963289f44b29/recovered.edits/0000000000000000004-kalashnikov%2C54919%2C1544681197805.1544681200975.temp (wrote 2 edits, skipped 0 edits in 13ms 2018-12-12 22:06:45,265 INFO [split-log-closeStream-1] wal.WALSplitter$LogRecoveredEditsOutputSink(1423): Rename hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/hbase/namespace/b61e07ebe62dcc1570e2963289f44b29/recovered.edits/0000000000000000004-kalashnikov%2C54919%2C1544681197805.1544681200975.temp to hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/hbase/namespace/b61e07ebe62dcc1570e2963289f44b29/recovered.edits/0000000000000000005 2018-12-12 22:06:45,265 INFO [RS_LOG_REPLAY_OPS-regionserver/kalashnikov:0-0] wal.WALSplitter(356): Processed 2 edits across 1 regions; edits skipped=2; log file=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/WALs/kalashnikov,54919,1544681197805-splitting/kalashnikov%2C54919%2C1544681197805.1544681200975, length=822, corrupted=false, progress failed=false 2018-12-12 22:06:45,268 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:54915-0x167a62c89310000, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fkalashnikov%2C54919%2C1544681197805-splitting%2Fkalashnikov%252C54919%252C1544681197805.1544681200975 2018-12-12 22:06:45,268 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:54917-0x167a62c89310001, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fkalashnikov%2C54919%2C1544681197805-splitting%2Fkalashnikov%252C54919%252C1544681197805.1544681200975 2018-12-12 22:06:45,268 INFO [RS_LOG_REPLAY_OPS-regionserver/kalashnikov:0-0] coordination.ZkSplitLogWorkerCoordination(574): successfully transitioned task /hbase/splitWAL/WALs%2Fkalashnikov%2C54919%2C1544681197805-splitting%2Fkalashnikov%252C54919%252C1544681197805.1544681200975 to final state DONE kalashnikov,54917,1544681197760 2018-12-12 22:06:45,268 INFO [RS_LOG_REPLAY_OPS-regionserver/kalashnikov:0-0] handler.WALSplitterHandler(99): Worker kalashnikov,54917,1544681197760 done with task org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$ZkSplitTaskDetails@68009ecb in 106ms. Status = DONE 2018-12-12 22:06:45,268 INFO [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination(372): Task /hbase/splitWAL/WALs%2Fkalashnikov%2C54919%2C1544681197805-splitting%2Fkalashnikov%252C54919%252C1544681197805.1544681200975 entered state=DONE kalashnikov,54917,1544681197760 2018-12-12 22:06:45,273 INFO [Time-limited test-EventThread] wal.WALSplitter(447): Archived processed log hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/WALs/kalashnikov,54919,1544681197805-splitting/kalashnikov%2C54919%2C1544681197805.1544681200975 to hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/oldWALs/kalashnikov%2C54919%2C1544681197805.1544681200975 2018-12-12 22:06:45,275 INFO [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination(415): Done splitting /hbase/splitWAL/WALs%2Fkalashnikov%2C54919%2C1544681197805-splitting%2Fkalashnikov%252C54919%252C1544681197805.1544681200975 2018-12-12 22:06:45,277 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:54915-0x167a62c89310000, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fkalashnikov%2C54919%2C1544681197805-splitting%2Fkalashnikov%252C54919%252C1544681197805.1544681200975 2018-12-12 22:06:45,277 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:54917-0x167a62c89310001, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitWAL/WALs%2Fkalashnikov%2C54919%2C1544681197805-splitting%2Fkalashnikov%252C54919%252C1544681197805.1544681200975 2018-12-12 22:06:45,277 DEBUG [Time-limited test-EventThread] coordination.ZKSplitLogManagerCoordination$DeleteAsyncCallback(696): Deleted /hbase/splitWAL/WALs%2Fkalashnikov%2C54919%2C1544681197805-splitting%2Fkalashnikov%252C54919%252C1544681197805.1544681200975 2018-12-12 22:06:45,277 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:54917-0x167a62c89310001, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitWAL 2018-12-12 22:06:45,280 WARN [PEWorker-9] master.SplitLogManager(283): Returning success without actually splitting and deleting all the log files in path hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/WALs/kalashnikov,54919,1544681197805-splitting: [FileStatus{path=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/WALs/kalashnikov,54919,1544681197805-splitting/kalashnikov%2C54919%2C1544681197805.meta.1544681201269.meta; isDirectory=false; length=3800; replication=2; blocksize=268435456; modification_time=1544681204985; access_time=1544681201270; owner=stack.hfs.1; group=supergroup; permission=rw-r--r--; isSymlink=false}] org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.PathIsNotEmptyDirectoryException): `/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/WALs/kalashnikov,54919,1544681197805-splitting is non empty': Directory is not empty at org.apache.hadoop.hdfs.server.namenode.FSDirDeleteOp.delete(FSDirDeleteOp.java:84) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:3690) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.delete(NameNodeRpcServer.java:953) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.delete(ClientNamenodeProtocolServerSideTranslatorPB.java:623) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2211) at org.apache.hadoop.ipc.Client.call(Client.java:1476) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy27.delete(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:545) at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy30.delete(Unknown Source) at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy31.delete(Unknown Source) at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:372) at com.sun.proxy.$Proxy31.delete(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2053) at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:707) at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:703) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:703) at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:277) at org.apache.hadoop.hbase.master.MasterWalManager.splitLog(MasterWalManager.java:401) at org.apache.hadoop.hbase.master.MasterWalManager.splitLog(MasterWalManager.java:386) at org.apache.hadoop.hbase.master.MasterWalManager.splitLog(MasterWalManager.java:283) at org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.splitLogs(ServerCrashProcedure.java:253) at org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.executeFromState(ServerCrashProcedure.java:159) at org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.executeFromState(ServerCrashProcedure.java:59) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:189) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:965) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1723) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1462) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1200(ProcedureExecutor.java:78) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2039) 2018-12-12 22:06:45,281 INFO [PEWorker-9] master.SplitLogManager(297): Finished splitting (more than or equal to) 822 bytes in 1 log files in [hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/WALs/kalashnikov,54919,1544681197805-splitting] in 136ms 2018-12-12 22:06:45,283 DEBUG [PEWorker-9] master.MasterWalManager(436): Archived meta log hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/WALs/kalashnikov,54919,1544681197805-splitting/kalashnikov%2C54919%2C1544681197805.meta.1544681201269.meta to hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/oldWALs/kalashnikov%2C54919%2C1544681197805.meta.1544681201269.meta 2018-12-12 22:06:45,284 DEBUG [PEWorker-9] procedure.ServerCrashProcedure(258): Done splitting WALs pid=12, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true; ServerCrashProcedure server=kalashnikov,54919,1544681197805, splitWal=true, meta=false 2018-12-12 22:06:45,284 DEBUG [PEWorker-9] procedure2.RootProcedureState(153): Add procedure pid=12, state=RUNNABLE:SERVER_CRASH_ASSIGN, locked=true; ServerCrashProcedure server=kalashnikov,54919,1544681197805, splitWal=true, meta=false as the 2th rollback step 2018-12-12 22:06:45,286 DEBUG [PEWorker-9] procedure.MasterProcedureScheduler(356): Add ServerQueue(kalashnikov,54919,1544681197805, xlock=false sharedLock=0 size=0) to run queue because: pid=12, state=RUNNABLE:SERVER_CRASH_ASSIGN; ServerCrashProcedure server=kalashnikov,54919,1544681197805, splitWal=true, meta=false released exclusive lock 2018-12-12 22:06:45,286 DEBUG [PEWorker-9] procedure.MasterProcedureScheduler(356): Add ServerQueue(kalashnikov,54919,1544681197805, xlock=false sharedLock=0 size=1) to run queue because: the exclusive lock is not held by anyone when adding pid=12, state=RUNNABLE:SERVER_CRASH_ASSIGN; ServerCrashProcedure server=kalashnikov,54919,1544681197805, splitWal=true, meta=false 2018-12-12 22:06:45,286 DEBUG [PEWorker-9] procedure.MasterProcedureScheduler(366): Remove ServerQueue(kalashnikov,54919,1544681197805, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=12, state=RUNNABLE:SERVER_CRASH_ASSIGN; ServerCrashProcedure server=kalashnikov,54919,1544681197805, splitWal=true, meta=false 2018-12-12 22:06:45,286 DEBUG [PEWorker-9] procedure.MasterProcedureScheduler(366): Remove ServerQueue(kalashnikov,54919,1544681197805, xlock=true (12) sharedLock=0 size=0) from run queue because: pid=12, state=RUNNABLE:SERVER_CRASH_ASSIGN; ServerCrashProcedure server=kalashnikov,54919,1544681197805, splitWal=true, meta=false held exclusive lock 2018-12-12 22:06:45,288 INFO [PEWorker-9] procedure2.ProcedureExecutor(1758): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=b61e07ebe62dcc1570e2963289f44b29}, {pid=14, ppid=12, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=test, region=30eb0f9efacc61d337395de976dcb330}] 2018-12-12 22:06:45,288 DEBUG [PEWorker-9] procedure2.RootProcedureState(153): Add procedure pid=12, state=WAITING:SERVER_CRASH_HANDLE_RIT2, locked=true; ServerCrashProcedure server=kalashnikov,54919,1544681197805, splitWal=true, meta=false as the 3th rollback step 2018-12-12 22:06:45,289 DEBUG [PEWorker-9] procedure.MasterProcedureScheduler(356): Add TableQueue(hbase:namespace, xlock=false sharedLock=0 size=1) to run queue because: the exclusive lock is not held by anyone when adding pid=13, ppid=12, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=b61e07ebe62dcc1570e2963289f44b29 2018-12-12 22:06:45,289 DEBUG [PEWorker-9] procedure.MasterProcedureScheduler(356): Add TableQueue(test, xlock=false sharedLock=0 size=1) to run queue because: the exclusive lock is not held by anyone when adding pid=14, ppid=12, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=test, region=30eb0f9efacc61d337395de976dcb330 2018-12-12 22:06:45,290 DEBUG [PEWorker-7] procedure.MasterProcedureScheduler(366): Remove TableQueue(hbase:namespace, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=13, ppid=12, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=b61e07ebe62dcc1570e2963289f44b29 2018-12-12 22:06:45,290 DEBUG [PEWorker-2] procedure.MasterProcedureScheduler(366): Remove TableQueue(test, xlock=false sharedLock=0 size=0) from run queue because: queue is empty after polling out pid=14, ppid=12, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=test, region=30eb0f9efacc61d337395de976dcb330 2018-12-12 22:06:45,290 INFO [PEWorker-2] procedure.MasterProcedureScheduler(741): Took xlock for pid=14, ppid=12, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=test, region=30eb0f9efacc61d337395de976dcb330 2018-12-12 22:06:45,290 INFO [PEWorker-7] procedure.MasterProcedureScheduler(741): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=b61e07ebe62dcc1570e2963289f44b29 2018-12-12 22:06:45,291 DEBUG [PEWorker-9] procedure.MasterProcedureScheduler(356): Add ServerQueue(kalashnikov,54919,1544681197805, xlock=false sharedLock=0 size=0) to run queue because: pid=12, state=WAITING:SERVER_CRASH_HANDLE_RIT2; ServerCrashProcedure server=kalashnikov,54919,1544681197805, splitWal=true, meta=false released exclusive lock 2018-12-12 22:06:45,291 INFO [PEWorker-2] assignment.AssignProcedure(254): Starting pid=14, ppid=12, state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure table=test, region=30eb0f9efacc61d337395de976dcb330; rit=OFFLINE, location=kalashnikov,54919,1544681197805; forceNewPlan=false, retain=true 2018-12-12 22:06:45,291 DEBUG [PEWorker-2] procedure2.RootProcedureState(153): Add procedure pid=14, ppid=12, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=test, region=30eb0f9efacc61d337395de976dcb330 as the 4th rollback step 2018-12-12 22:06:45,395 INFO [PEWorker-7] assignment.AssignProcedure(254): Starting pid=13, ppid=12, state=RUNNABLE:REGION_TRANSITION_QUEUE, locked=true; AssignProcedure table=hbase:namespace, region=b61e07ebe62dcc1570e2963289f44b29; rit=OFFLINE, location=kalashnikov,54919,1544681197805; forceNewPlan=false, retain=true 2018-12-12 22:06:45,395 DEBUG [PEWorker-7] procedure2.RootProcedureState(153): Add procedure pid=13, ppid=12, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:namespace, region=b61e07ebe62dcc1570e2963289f44b29 as the 5th rollback step 2018-12-12 22:06:45,396 DEBUG [master/kalashnikov:0] procedure.MasterProcedureScheduler(356): Add TableQueue(hbase:namespace, xlock=false sharedLock=1 size=1) to run queue because: pid=13, ppid=12, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:namespace, region=b61e07ebe62dcc1570e2963289f44b29 has lock 2018-12-12 22:06:45,396 DEBUG [master/kalashnikov:0] procedure.MasterProcedureScheduler(356): Add TableQueue(test, xlock=false sharedLock=1 size=1) to run queue because: pid=14, ppid=12, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=test, region=30eb0f9efacc61d337395de976dcb330 has lock 2018-12-12 22:06:45,396 DEBUG [PEWorker-16] procedure.MasterProcedureScheduler(366): Remove TableQueue(hbase:namespace, xlock=false sharedLock=1 size=0) from run queue because: queue is empty after polling out pid=13, ppid=12, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=hbase:namespace, region=b61e07ebe62dcc1570e2963289f44b29 2018-12-12 22:06:45,396 DEBUG [PEWorker-6] procedure.MasterProcedureScheduler(366): Remove TableQueue(test, xlock=false sharedLock=1 size=0) from run queue because: queue is empty after polling out pid=14, ppid=12, state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; AssignProcedure table=test, region=30eb0f9efacc61d337395de976dcb330 2018-12-12 22:06:45,396 INFO [PEWorker-6] assignment.RegionStateStore(200): pid=14 updating hbase:meta row=30eb0f9efacc61d337395de976dcb330, regionState=OPENING, regionLocation=kalashnikov,54917,1544681197760 2018-12-12 22:06:45,399 INFO [PEWorker-16] assignment.RegionStateStore(200): pid=13 updating hbase:meta row=b61e07ebe62dcc1570e2963289f44b29, regionState=OPENING, regionLocation=kalashnikov,54917,1544681197760 2018-12-12 22:06:45,848 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-12-12 22:06:45,849 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-12-12 22:06:45,849 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(51): Creating new MetricsTableSourceImpl for table 2018-12-12 22:06:45,868 INFO [regionserver/kalashnikov:0.Chore.1] hbase.ScheduledChore(180): Chore: CompactionChecker was stopped 2018-12-12 22:06:45,868 INFO [regionserver/kalashnikov:0.Chore.1] hbase.ScheduledChore(180): Chore: MemstoreFlusherChore was stopped 2018-12-12 22:06:45,883 INFO [regionserver/kalashnikov:0.leaseChecker] regionserver.Leases(149): Closed leases 2018-12-12 22:06:46,874 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-12-12 22:06:46,945 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(135): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-12-12 22:06:46,946 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(139): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2018-12-12 22:06:55,085 INFO [Time-limited test] hbase.ResourceChecker(172): after: regionserver.TestCleanupMetaWAL#testCleanupMetaWAL Thread=270 (was 299), OpenFileDescriptor=815 (was 832), MaxFileDescriptor=10240 (was 10240), SystemLoadAverage=290 (was 305), ProcessCount=483 (was 483), AvailableMemoryMB=650 (was 725) 2018-12-12 22:06:55,087 ERROR [Time-limited test] server.ZooKeeperServer(472): ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes 2018-12-12 22:06:55,088 INFO [Time-limited test] zookeeper.MiniZooKeeperCluster(326): Shutdown MiniZK cluster with all ZK servers 2018-12-12 22:06:55,121 INFO [Thread-2] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@104a1bb9 2018-12-12 22:06:55,121 INFO [Thread-2] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished. 2018-12-12 22:06:55,121 INFO [Thread-2] regionserver.ShutdownHook$ShutdownHookThread(112): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@104a1bb9 2018-12-12 22:06:55,121 INFO [Thread-2] regionserver.HRegionServer(2138): ***** STOPPING region server 'kalashnikov,54917,1544681197760' ***** 2018-12-12 22:06:55,121 INFO [Thread-2] regionserver.HRegionServer(2152): STOPPED: Shutdown hook 2018-12-12 22:06:55,121 INFO [RS:0;kalashnikov:54917] regionserver.SplitLogWorker(166): Sending interrupt to stop the worker thread 2018-12-12 22:06:55,121 INFO [RS:0;kalashnikov:54917] regionserver.HeapMemoryManager(221): Stopping 2018-12-12 22:06:55,121 INFO [SplitLogWorker-kalashnikov:54917] regionserver.SplitLogWorker(148): SplitLogWorker interrupted. Exiting. 2018-12-12 22:06:55,122 INFO [RS:0;kalashnikov:54917] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager gracefully. 2018-12-12 22:06:55,122 INFO [RS:0;kalashnikov:54917] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2018-12-12 22:06:55,122 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(382): MemStoreFlusher.0 exiting 2018-12-12 22:06:55,122 INFO [RS:0;kalashnikov:54917] regionserver.HRegionServer(1079): stopping server kalashnikov,54917,1544681197760 2018-12-12 22:06:55,122 DEBUG [RS:0;kalashnikov:54917] zookeeper.MetaTableLocator(642): Stopping MetaTableLocator 2018-12-12 22:06:55,122 INFO [SplitLogWorker-kalashnikov:54917] regionserver.SplitLogWorker(157): SplitLogWorker kalashnikov,54917,1544681197760 exiting 2018-12-12 22:06:55,122 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(382): MemStoreFlusher.1 exiting 2018-12-12 22:06:55,123 INFO [RS:0;kalashnikov:54917] zookeeper.ReadOnlyZKClient(350): Close zookeeper connection 0x2f9816cf to localhost:63470 2018-12-12 22:06:55,123 DEBUG [RS:0;kalashnikov:54917] ipc.AbstractRpcClient(483): Stopping rpc client 2018-12-12 22:06:55,123 INFO [RS:0;kalashnikov:54917] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-12-12 22:06:55,123 INFO [RS:0;kalashnikov:54917] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-12-12 22:06:55,123 INFO [RS:0;kalashnikov:54917] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-12-12 22:06:55,123 INFO [RS:0;kalashnikov:54917] regionserver.HRegionServer(1384): Waiting on 1 regions to close 2018-12-12 22:06:55,124 DEBUG [RS:0;kalashnikov:54917] regionserver.HRegionServer(1388): Online Regions={1588230740=hbase:meta,,1.1588230740} 2018-12-12 22:06:55,125 DEBUG [RS_CLOSE_META-regionserver/kalashnikov:0-0] regionserver.HRegion(1541): Closing 1588230740, disabling compactions & flushes 2018-12-12 22:06:55,125 DEBUG [RS_CLOSE_META-regionserver/kalashnikov:0-0] regionserver.HRegion(1581): Updates disabled for region hbase:meta,,1.1588230740 2018-12-12 22:06:55,136 DEBUG [RS_CLOSE_META-regionserver/kalashnikov:0-0] wal.WALSplitter(695): Wrote file=hdfs://localhost:54901/user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/hbase/meta/1588230740/recovered.edits/20.seqid, newMaxSeqId=20, maxSeqId=17 2018-12-12 22:06:55,138 DEBUG [RS_CLOSE_META-regionserver/kalashnikov:0-0] coprocessor.CoprocessorHost(288): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-12-12 22:06:55,139 INFO [RS_CLOSE_META-regionserver/kalashnikov:0-0] regionserver.HRegion(1698): Closed hbase:meta,,1.1588230740 2018-12-12 22:06:55,139 DEBUG [RS_CLOSE_META-regionserver/kalashnikov:0-0] handler.CloseRegionHandler(124): Closed hbase:meta,,1.1588230740 2018-12-12 22:06:55,188 DEBUG [master/kalashnikov:0:becomeActiveMaster-EventThread] zookeeper.ZKWatcher(478): replicationLogCleaner-0x167a62c89310006, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2018-12-12 22:06:55,189 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:54917-0x167a62c89310001, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2018-12-12 22:06:55,189 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(548): regionserver:54917-0x167a62c89310001, quorum=localhost:63470, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2018-12-12 22:06:55,189 DEBUG [master/kalashnikov:0:becomeActiveMaster-EventThread] zookeeper.ZKWatcher(548): replicationLogCleaner-0x167a62c89310006, quorum=localhost:63470, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2018-12-12 22:06:55,188 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:54915-0x167a62c89310000, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2018-12-12 22:06:55,189 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(548): master:54915-0x167a62c89310000, quorum=localhost:63470, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2018-12-12 22:06:55,189 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): regionserver:54917-0x167a62c89310001, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2018-12-12 22:06:55,190 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(548): regionserver:54917-0x167a62c89310001, quorum=localhost:63470, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2018-12-12 22:06:55,189 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(478): master:54915-0x167a62c89310000, quorum=localhost:63470, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2018-12-12 22:06:55,190 DEBUG [Time-limited test-EventThread] zookeeper.ZKWatcher(548): master:54915-0x167a62c89310000, quorum=localhost:63470, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2018-12-12 22:06:55,328 INFO [RS:0;kalashnikov:54917] regionserver.HRegionServer(1107): stopping server kalashnikov,54917,1544681197760; all regions closed. 2018-12-12 22:06:55,336 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54906 is added to blk_1073741837_1013{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b8b11ce4-1484-4801-8b0b-e3a38c78a69a:NORMAL:127.0.0.1:54902|RBW], ReplicaUC[[DISK]DS-062f61d8-8f41-4a9e-a2ce-6913541fccba:NORMAL:127.0.0.1:54906|RBW]]} size 0 2018-12-12 22:06:55,337 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54902 is added to blk_1073741837_1013{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b8b11ce4-1484-4801-8b0b-e3a38c78a69a:NORMAL:127.0.0.1:54902|RBW], ReplicaUC[[DISK]DS-062f61d8-8f41-4a9e-a2ce-6913541fccba:NORMAL:127.0.0.1:54906|RBW]]} size 0 2018-12-12 22:06:55,341 DEBUG [RS:0;kalashnikov:54917] wal.AbstractFSWAL(847): Moved 1 WAL file(s) to /user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/oldWALs 2018-12-12 22:06:55,341 INFO [RS:0;kalashnikov:54917] wal.AbstractFSWAL(850): Closed WAL: AsyncFSWAL kalashnikov%2C54917%2C1544681197760.meta:.meta(num 1544681204002) 2018-12-12 22:06:55,348 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54902 is added to blk_1073741830_1006{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b8b11ce4-1484-4801-8b0b-e3a38c78a69a:NORMAL:127.0.0.1:54902|RBW], ReplicaUC[[DISK]DS-062f61d8-8f41-4a9e-a2ce-6913541fccba:NORMAL:127.0.0.1:54906|RBW]]} size 0 2018-12-12 22:06:55,348 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54906 is added to blk_1073741830_1006{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b8b11ce4-1484-4801-8b0b-e3a38c78a69a:NORMAL:127.0.0.1:54902|RBW], ReplicaUC[[DISK]DS-062f61d8-8f41-4a9e-a2ce-6913541fccba:NORMAL:127.0.0.1:54906|RBW]]} size 0 2018-12-12 22:06:55,352 DEBUG [RS:0;kalashnikov:54917] wal.AbstractFSWAL(847): Moved 1 WAL file(s) to /user/stack/test-data/263f0937-a35e-bab1-1f0a-77a2b5d69524/oldWALs 2018-12-12 22:06:55,352 INFO [RS:0;kalashnikov:54917] wal.AbstractFSWAL(850): Closed WAL: AsyncFSWAL kalashnikov%2C54917%2C1544681197760:(num 1544681200975) 2018-12-12 22:06:55,352 DEBUG [RS:0;kalashnikov:54917] ipc.AbstractRpcClient(483): Stopping rpc client 2018-12-12 22:06:55,353 INFO [RS:0;kalashnikov:54917] regionserver.Leases(149): Closed leases 2018-12-12 22:06:55,353 INFO [RS:0;kalashnikov:54917] hbase.ChoreService(327): Chore service for: regionserver/kalashnikov:0 had [[ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS]] on shutdown 2018-12-12 22:06:55,353 INFO [regionserver/kalashnikov:0.logRoller] regionserver.LogRoller(212): LogRoller exiting. 2018-12-12 22:06:55,354 INFO [RS:0;kalashnikov:54917] ipc.NettyRpcServer(144): Stopping server on /192.168.1.137:54917 2018-12-12 22:06:55,403 DEBUG [RS-EventLoopGroup-4-7] ipc.FailedServers(56): Added failed server with address kalashnikov/192.168.1.137:54919 to list caused by org.apache.hbase.thirdparty.io.netty.channel.ConnectTimeoutException: connection timed out: kalashnikov/192.168.1.137:54919 2018-12-12 22:06:55,917 INFO [regionserver/kalashnikov:0.leaseChecker] regionserver.Leases(149): Closed leases 2018-12-12 22:06:56,708 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(157): Removing adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-12-12 22:06:56,722 WARN [ReadOnlyZKClient-localhost:63470@0x6d7efced] zookeeper.ReadOnlyZKClient$ZKTask$1(189): 0x6d7efced to localhost:63470 failed for get of /hbase/meta-region-server, code = CONNECTIONLOSS, retries = 1 2018-12-12 22:06:57,269 DEBUG [RS:0;kalashnikov:54917] zookeeper.RecoverableZooKeeper(263): Retry, connectivity issue (JVM Pause?); quorum=localhost:63470,exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/rs/kalashnikov,54917,1544681197760 2018-12-12 22:06:58,100 WARN [ReadOnlyZKClient-localhost:63470@0x6d7efced] zookeeper.ReadOnlyZKClient$ZKTask$1(189): 0x6d7efced to localhost:63470 failed for get of /hbase/meta-region-server, code = CONNECTIONLOSS, retries = 2 2018-12-12 22:06:58,746 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(2147): Worker stuck PEWorker-6(pid=14), run time 13.3500sec 2018-12-12 22:06:58,747 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(2147): Worker stuck PEWorker-16(pid=13), run time 13.3500sec 2018-12-12 22:06:59,119 DEBUG [RS:0;kalashnikov:54917] zookeeper.RecoverableZooKeeper(263): Retry, connectivity issue (JVM Pause?); quorum=localhost:63470,exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/rs/kalashnikov,54917,1544681197760 2018-12-12 22:07:00,017 WARN [ReadOnlyZKClient-localhost:63470@0x6d7efced] zookeeper.ReadOnlyZKClient$ZKTask$1(189): 0x6d7efced to localhost:63470 failed for get of /hbase/meta-region-server, code = CONNECTIONLOSS, retries = 3 2018-12-12 22:07:00,643 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-12-12 22:07:01,486 WARN [ReadOnlyZKClient-localhost:63470@0x6d7efced] zookeeper.ReadOnlyZKClient$ZKTask$1(189): 0x6d7efced to localhost:63470 failed for get of /hbase/meta-region-server, code = CONNECTIONLOSS, retries = 4 2018-12-12 22:07:02,638 DEBUG [RS:0;kalashnikov:54917] zookeeper.RecoverableZooKeeper(263): Retry, connectivity issue (JVM Pause?); quorum=localhost:63470,exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/rs/kalashnikov,54917,1544681197760 2018-12-12 22:07:03,342 WARN [ReadOnlyZKClient-localhost:63470@0x6d7efced] zookeeper.ReadOnlyZKClient$ZKTask$1(189): 0x6d7efced to localhost:63470 failed for get of /hbase/meta-region-server, code = CONNECTIONLOSS, retries = 5 2018-12-12 22:07:03,751 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(2147): Worker stuck PEWorker-6(pid=14), run time 18.3550sec 2018-12-12 22:07:03,752 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(2147): Worker stuck PEWorker-16(pid=13), run time 18.3560sec 2018-12-12 22:07:05,446 WARN [ReadOnlyZKClient-localhost:63470@0x6d7efced] zookeeper.ReadOnlyZKClient$ZKTask$1(189): 0x6d7efced to localhost:63470 failed for get of /hbase/meta-region-server, code = CONNECTIONLOSS, retries = 6 2018-12-12 22:07:07,100 WARN [ReadOnlyZKClient-localhost:63470@0x6d7efced] zookeeper.ReadOnlyZKClient$ZKTask$1(189): 0x6d7efced to localhost:63470 failed for get of /hbase/meta-region-server, code = CONNECTIONLOSS, retries = 7 2018-12-12 22:07:07,232 DEBUG [RS:0;kalashnikov:54917] zookeeper.RecoverableZooKeeper(263): Retry, connectivity issue (JVM Pause?); quorum=localhost:63470,exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/rs/kalashnikov,54917,1544681197760 2018-12-12 22:07:08,850 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(2147): Worker stuck PEWorker-6(pid=14), run time 23.4540sec 2018-12-12 22:07:08,850 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(2147): Worker stuck PEWorker-16(pid=13), run time 23.4540sec 2018-12-12 22:07:09,130 WARN [ReadOnlyZKClient-localhost:63470@0x6d7efced] zookeeper.ReadOnlyZKClient$ZKTask$1(189): 0x6d7efced to localhost:63470 failed for get of /hbase/meta-region-server, code = CONNECTIONLOSS, retries = 8 2018-12-12 22:07:10,317 WARN [ReadOnlyZKClient-localhost:63470@0x6d7efced] zookeeper.ReadOnlyZKClient$ZKTask$1(189): 0x6d7efced to localhost:63470 failed for get of /hbase/meta-region-server, code = CONNECTIONLOSS, retries = 9 2018-12-12 22:07:11,666 WARN [ReadOnlyZKClient-localhost:63470@0x6d7efced] zookeeper.ReadOnlyZKClient$ZKTask$1(189): 0x6d7efced to localhost:63470 failed for get of /hbase/meta-region-server, code = CONNECTIONLOSS, retries = 10 2018-12-12 22:07:13,598 WARN [ReadOnlyZKClient-localhost:63470@0x6d7efced] zookeeper.ReadOnlyZKClient$ZKTask$1(189): 0x6d7efced to localhost:63470 failed for get of /hbase/meta-region-server, code = CONNECTIONLOSS, retries = 11 2018-12-12 22:07:13,852 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(2147): Worker stuck PEWorker-6(pid=14), run time 28.4560sec 2018-12-12 22:07:13,852 WARN [ProcExecTimeout] procedure2.ProcedureExecutor$WorkerMonitor(2147): Worker stuck PEWorker-16(pid=13), run time 28.4560sec 2018-12-12 22:07:15,238 WARN [ReadOnlyZKClient-localhost:63470@0x6d7efced] zookeeper.ReadOnlyZKClient$ZKTask$1(189): 0x6d7efced to localhost:63470 failed for get of /hbase/meta-region-server, code = CONNECTIONLOSS, retries = 12 2018-12-12 22:07:15,779 ERROR [RS:0;kalashnikov:54917] zookeeper.RecoverableZooKeeper(259): ZooKeeper delete failed after 4 attempts 2018-12-12 22:07:15,779 WARN [RS:0;kalashnikov:54917] regionserver.HRegionServer(1144): Failed deleting my ephemeral node org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/rs/kalashnikov,54917,1544681197760 at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:166) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1231) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1220) at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1587) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1141) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:183) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:129) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:164) at java.lang.Thread.run(Thread.java:748) 2018-12-12 22:07:16,622 INFO [RS:0;kalashnikov:54917] regionserver.HRegionServer(1154): Exiting; stopping=kalashnikov,54917,1544681197760; zookeeper connection closed. 2018-12-12 22:07:16,622 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@6d1f5718] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(221): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@6d1f5718 2018-12-12 22:07:16,623 INFO [Thread-2] regionserver.ShutdownHook$ShutdownHookThread(121): Starting fs shutdown hook thread. 2018-12-12 22:07:16,639 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54902 is added to blk_1073741841_1017{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-062f61d8-8f41-4a9e-a2ce-6913541fccba:NORMAL:127.0.0.1:54906|RBW], ReplicaUC[[DISK]DS-b8b11ce4-1484-4801-8b0b-e3a38c78a69a:NORMAL:127.0.0.1:54902|FINALIZED]]} size 0 2018-12-12 22:07:16,640 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54906 is added to blk_1073741841_1017 size 198 2018-12-12 22:07:16,645 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54906 is added to blk_1073741829_1005{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b8b11ce4-1484-4801-8b0b-e3a38c78a69a:NORMAL:127.0.0.1:54902|RBW], ReplicaUC[[DISK]DS-062f61d8-8f41-4a9e-a2ce-6913541fccba:NORMAL:127.0.0.1:54906|RBW]]} size 211 2018-12-12 22:07:16,646 INFO [Block report processor] blockmanagement.BlockManager(2645): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:54902 is added to blk_1073741829_1005{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-b8b11ce4-1484-4801-8b0b-e3a38c78a69a:NORMAL:127.0.0.1:54902|RBW], ReplicaUC[[DISK]DS-062f61d8-8f41-4a9e-a2ce-6913541fccba:NORMAL:127.0.0.1:54906|RBW]]} size 211 2018-12-12 22:07:16,648 INFO [Thread-2] regionserver.ShutdownHook$ShutdownHookThread(135): Shutdown hook finished.