2018-01-13 19:32:06,145 INFO [main] hbase.HBaseTestingUtility(945): Starting up minicluster with 1 master(s) and 3 regionserver(s) and 3 datanode(s) 2018-01-13 19:32:06,153 INFO [main] hbase.HBaseZKTestingUtility(86): Created new mini-cluster data directory: /home/zhangduo/hbase/code/hbase-server/target/test-data/83985523-c805-46ca-b109-55b9bc1b0e3a/cluster_6265c6a0-6881-4ad9-aaf9-0fd90cf9199c, deleteOnExit=true 2018-01-13 19:32:06,153 INFO [main] hbase.HBaseTestingUtility(960): STARTING DFS 2018-01-13 19:32:06,154 INFO [main] hbase.HBaseTestingUtility(737): Setting test.cache.data to /home/zhangduo/hbase/code/hbase-server/target/test-data/83985523-c805-46ca-b109-55b9bc1b0e3a/cache_data in system properties and HBase conf 2018-01-13 19:32:06,154 INFO [main] hbase.HBaseTestingUtility(737): Setting hadoop.tmp.dir to /home/zhangduo/hbase/code/hbase-server/target/test-data/83985523-c805-46ca-b109-55b9bc1b0e3a/hadoop_tmp in system properties and HBase conf 2018-01-13 19:32:06,154 INFO [main] hbase.HBaseTestingUtility(737): Setting hadoop.log.dir to /home/zhangduo/hbase/code/hbase-server/target/test-data/83985523-c805-46ca-b109-55b9bc1b0e3a/hadoop_logs in system properties and HBase conf 2018-01-13 19:32:06,154 INFO [main] hbase.HBaseTestingUtility(737): Setting mapreduce.cluster.local.dir to /home/zhangduo/hbase/code/hbase-server/target/test-data/83985523-c805-46ca-b109-55b9bc1b0e3a/mapred_local in system properties and HBase conf 2018-01-13 19:32:06,155 INFO [main] hbase.HBaseTestingUtility(737): Setting mapreduce.cluster.temp.dir to /home/zhangduo/hbase/code/hbase-server/target/test-data/83985523-c805-46ca-b109-55b9bc1b0e3a/mapred_temp in system properties and HBase conf 2018-01-13 19:32:06,155 INFO [main] hbase.HBaseTestingUtility(728): read short circuit is OFF 2018-01-13 19:32:06,246 WARN [main] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-01-13 19:32:06,549 DEBUG [main] fs.HFileSystem(318): The file system is not a DistributedFileSystem. Skipping on block location reordering Formatting using clusterid: testClusterID 2018-01-13 19:32:07,566 WARN [main] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2018-01-13 19:32:07,777 INFO [main] log.Slf4jLog(67): Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2018-01-13 19:32:07,847 INFO [main] log.Slf4jLog(67): jetty-6.1.26 2018-01-13 19:32:07,886 INFO [main] log.Slf4jLog(67): Extract jar:file:/home/zhangduo/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.7.4/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to /tmp/Jetty_localhost_44485_hdfs____.s5a36t/webapp 2018-01-13 19:32:09,952 INFO [main] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44485 2018-01-13 19:32:10,456 INFO [main] log.Slf4jLog(67): jetty-6.1.26 2018-01-13 19:32:10,462 INFO [main] log.Slf4jLog(67): Extract jar:file:/home/zhangduo/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.7.4/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to /tmp/Jetty_localhost_45403_datanode____.expczn/webapp 2018-01-13 19:32:11,436 INFO [main] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45403 2018-01-13 19:32:11,760 INFO [main] log.Slf4jLog(67): jetty-6.1.26 2018-01-13 19:32:11,767 INFO [main] log.Slf4jLog(67): Extract jar:file:/home/zhangduo/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.7.4/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to /tmp/Jetty_localhost_45407_datanode____.ycfxtr/webapp 2018-01-13 19:32:12,120 ERROR [DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/83985523-c805-46ca-b109-55b9bc1b0e3a/cluster_6265c6a0-6881-4ad9-aaf9-0fd90cf9199c/dfs/data/data1/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/83985523-c805-46ca-b109-55b9bc1b0e3a/cluster_6265c6a0-6881-4ad9-aaf9-0fd90cf9199c/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:44545] datanode.DirectoryScanner(430): dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000 2018-01-13 19:32:12,339 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x4ddaaad5ae14: from storage DS-3a12d4a8-59eb-4588-98a1-23de755fb453 node DatanodeRegistration(127.0.0.1:35477, datanodeUuid=6e6a28cb-4d66-4070-96ce-b0e3d82a40ef, infoPort=39899, infoSecurePort=0, ipcPort=37175, storageInfo=lv=-56;cid=testClusterID;nsid=857557072;c=0), blocks: 0, hasStaleStorage: true, processing time: 2 msecs 2018-01-13 19:32:12,340 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x4ddaaad5ae14: from storage DS-7b74f716-787f-414f-bbbf-7d604b248220 node DatanodeRegistration(127.0.0.1:35477, datanodeUuid=6e6a28cb-4d66-4070-96ce-b0e3d82a40ef, infoPort=39899, infoSecurePort=0, ipcPort=37175, storageInfo=lv=-56;cid=testClusterID;nsid=857557072;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs 2018-01-13 19:32:13,140 INFO [main] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45407 2018-01-13 19:32:13,255 INFO [main] log.Slf4jLog(67): jetty-6.1.26 2018-01-13 19:32:13,276 INFO [main] log.Slf4jLog(67): Extract jar:file:/home/zhangduo/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.7.4/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to /tmp/Jetty_localhost_42675_datanode____.7z237n/webapp 2018-01-13 19:32:13,305 ERROR [DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/83985523-c805-46ca-b109-55b9bc1b0e3a/cluster_6265c6a0-6881-4ad9-aaf9-0fd90cf9199c/dfs/data/data3/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/83985523-c805-46ca-b109-55b9bc1b0e3a/cluster_6265c6a0-6881-4ad9-aaf9-0fd90cf9199c/dfs/data/data4/]] heartbeating to localhost/127.0.0.1:44545] datanode.DirectoryScanner(430): dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000 2018-01-13 19:32:13,320 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x4ddae5de87e3: from storage DS-c786ffef-0cfe-40db-9609-e37ce51fcb98 node DatanodeRegistration(127.0.0.1:39981, datanodeUuid=3496a9e7-5d22-43aa-a6bb-1b049f6249de, infoPort=32773, infoSecurePort=0, ipcPort=43437, storageInfo=lv=-56;cid=testClusterID;nsid=857557072;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs 2018-01-13 19:32:13,322 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x4ddae5de87e3: from storage DS-2682b908-d71d-4edd-a7d6-c00e22c135f0 node DatanodeRegistration(127.0.0.1:39981, datanodeUuid=3496a9e7-5d22-43aa-a6bb-1b049f6249de, infoPort=32773, infoSecurePort=0, ipcPort=43437, storageInfo=lv=-56;cid=testClusterID;nsid=857557072;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs 2018-01-13 19:32:14,222 INFO [main] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42675 2018-01-13 19:32:14,370 ERROR [DataNode: [[[DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/83985523-c805-46ca-b109-55b9bc1b0e3a/cluster_6265c6a0-6881-4ad9-aaf9-0fd90cf9199c/dfs/data/data5/, [DISK]file:/home/zhangduo/hbase/code/hbase-server/target/test-data/83985523-c805-46ca-b109-55b9bc1b0e3a/cluster_6265c6a0-6881-4ad9-aaf9-0fd90cf9199c/dfs/data/data6/]] heartbeating to localhost/127.0.0.1:44545] datanode.DirectoryScanner(430): dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000 2018-01-13 19:32:14,383 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x4ddb2529da3d: from storage DS-2ae3c181-7033-4bb1-926b-79afb8ba9668 node DatanodeRegistration(127.0.0.1:45841, datanodeUuid=8c8ddff3-9221-49b2-99ef-655fbc3368f3, infoPort=46553, infoSecurePort=0, ipcPort=45023, storageInfo=lv=-56;cid=testClusterID;nsid=857557072;c=0), blocks: 0, hasStaleStorage: true, processing time: 5 msecs 2018-01-13 19:32:14,384 INFO [Block report processor] blockmanagement.BlockManager(1933): BLOCK* processReport 0x4ddb2529da3d: from storage DS-6e234590-c709-4272-a78c-d0a2418a7171 node DatanodeRegistration(127.0.0.1:45841, datanodeUuid=8c8ddff3-9221-49b2-99ef-655fbc3368f3, infoPort=46553, infoSecurePort=0, ipcPort=45023, storageInfo=lv=-56;cid=testClusterID;nsid=857557072;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs 2018-01-13 19:32:14,669 ERROR [main] server.ZooKeeperServer(472): ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes 2018-01-13 19:32:14,680 INFO [main] zookeeper.MiniZooKeeperCluster(281): Started MiniZooKeeperCluster and ran successful 'stat' on client port=53209 2018-01-13 19:32:14,688 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 19:32:14,690 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 19:32:15,048 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39981 is added to blk_1073741825_1001{UCState=COMMITTED, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-3a12d4a8-59eb-4588-98a1-23de755fb453:NORMAL:127.0.0.1:35477|RBW], ReplicaUC[[DISK]DS-c786ffef-0cfe-40db-9609-e37ce51fcb98:NORMAL:127.0.0.1:39981|RBW], ReplicaUC[[DISK]DS-2ae3c181-7033-4bb1-926b-79afb8ba9668:NORMAL:127.0.0.1:45841|RBW]]} size 7 2018-01-13 19:32:15,048 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45841 is added to blk_1073741825_1001 size 7 2018-01-13 19:32:15,048 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35477 is added to blk_1073741825_1001 size 7 2018-01-13 19:32:15,482 INFO [main] util.FSUtils(513): Created version file at hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433 with version=8 2018-01-13 19:32:15,483 INFO [main] hbase.HBaseTestingUtility(1204): Setting hbase.fs.tmp.dir to hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/hbase-staging 2018-01-13 19:32:15,643 INFO [main] metrics.MetricRegistriesLoader(66): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2018-01-13 19:32:15,812 INFO [main] client.ConnectionUtils(122): master/ubuntu/127.0.1.1:0 server-side Connection retries=45 2018-01-13 19:32:15,830 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 19:32:15,831 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 19:32:15,832 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 19:32:15,921 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 19:32:16,076 DEBUG [main] util.ClassSize(229): Using Unsafe to estimate memory layout 2018-01-13 19:32:16,133 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/127.0.1.1:36079 2018-01-13 19:32:16,141 INFO [main] hfile.CacheConfig(553): Allocating On heap LruBlockCache size=1.04 GB, blockSize=64 KB 2018-01-13 19:32:16,146 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=820320, freeSize=1117800608, maxSize=1118620928, heapSize=820320, minSize=1062689856, minFactor=0.95, multiSize=531344928, multiFactor=0.5, singleSize=265672464, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 19:32:16,146 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=820320, freeSize=1117800608, maxSize=1118620928, heapSize=820320, minSize=1062689856, minFactor=0.95, multiSize=531344928, multiFactor=0.5, singleSize=265672464, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 19:32:16,148 DEBUG [main] mob.MobFileCache(123): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2018-01-13 19:32:16,150 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 19:32:16,152 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 19:32:16,189 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=master:36079 connecting to ZooKeeper ensemble=localhost:53209 2018-01-13 19:32:16,223 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:360790x0, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 19:32:16,224 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): master:36079-0x160ef4aa4c30000 connected 2018-01-13 19:32:16,264 DEBUG [main] zookeeper.ZKUtil(357): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 19:32:16,265 DEBUG [main] zookeeper.ZKUtil(357): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 19:32:16,425 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=36079 2018-01-13 19:32:16,428 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=36079 2018-01-13 19:32:16,432 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=36079 2018-01-13 19:32:16,436 INFO [main] master.HMaster(480): hbase.rootdir=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433, hbase.cluster.distributed=false 2018-01-13 19:32:16,539 INFO [main] client.ConnectionUtils(122): regionserver/ubuntu/127.0.1.1:0 server-side Connection retries=45 2018-01-13 19:32:16,539 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 19:32:16,539 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 19:32:16,540 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 19:32:16,545 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 19:32:16,546 INFO [main] io.ByteBufferPool(83): Created ByteBufferPool with bufferSize : 65536 and maxPoolSize : 320 2018-01-13 19:32:16,549 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/127.0.1.1:44139 2018-01-13 19:32:16,549 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=820320, freeSize=1117800608, maxSize=1118620928, heapSize=820320, minSize=1062689856, minFactor=0.95, multiSize=531344928, multiFactor=0.5, singleSize=265672464, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 19:32:16,550 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=820320, freeSize=1117800608, maxSize=1118620928, heapSize=820320, minSize=1062689856, minFactor=0.95, multiSize=531344928, multiFactor=0.5, singleSize=265672464, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 19:32:16,551 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 19:32:16,554 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 19:32:16,556 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:44139 connecting to ZooKeeper ensemble=localhost:53209 2018-01-13 19:32:16,576 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:441390x0, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 19:32:16,577 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): regionserver:44139-0x160ef4aa4c30001 connected 2018-01-13 19:32:16,578 DEBUG [main] zookeeper.ZKUtil(357): regionserver:44139-0x160ef4aa4c30001, quorum=localhost:53209, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 19:32:16,579 DEBUG [main] zookeeper.ZKUtil(357): regionserver:44139-0x160ef4aa4c30001, quorum=localhost:53209, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 19:32:16,579 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=44139 2018-01-13 19:32:16,582 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=44139 2018-01-13 19:32:16,587 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=44139 2018-01-13 19:32:16,613 INFO [main] client.ConnectionUtils(122): regionserver/ubuntu/127.0.1.1:0 server-side Connection retries=45 2018-01-13 19:32:16,613 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 19:32:16,614 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 19:32:16,614 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 19:32:16,614 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 19:32:16,614 INFO [main] io.ByteBufferPool(83): Created ByteBufferPool with bufferSize : 65536 and maxPoolSize : 320 2018-01-13 19:32:16,616 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/127.0.1.1:40283 2018-01-13 19:32:16,617 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=820320, freeSize=1117800608, maxSize=1118620928, heapSize=820320, minSize=1062689856, minFactor=0.95, multiSize=531344928, multiFactor=0.5, singleSize=265672464, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 19:32:16,617 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=820320, freeSize=1117800608, maxSize=1118620928, heapSize=820320, minSize=1062689856, minFactor=0.95, multiSize=531344928, multiFactor=0.5, singleSize=265672464, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 19:32:16,619 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 19:32:16,621 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 19:32:16,624 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:40283 connecting to ZooKeeper ensemble=localhost:53209 2018-01-13 19:32:16,641 DEBUG [main] zookeeper.ZKUtil(357): regionserver:402830x0, quorum=localhost:53209, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 19:32:16,641 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:402830x0, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 19:32:16,642 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): regionserver:40283-0x160ef4aa4c30002 connected 2018-01-13 19:32:16,643 DEBUG [main] zookeeper.ZKUtil(357): regionserver:40283-0x160ef4aa4c30002, quorum=localhost:53209, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 19:32:16,643 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=40283 2018-01-13 19:32:16,647 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=40283 2018-01-13 19:32:16,648 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=40283 2018-01-13 19:32:16,670 INFO [main] client.ConnectionUtils(122): regionserver/ubuntu/127.0.1.1:0 server-side Connection retries=45 2018-01-13 19:32:16,670 INFO [main] ipc.RpcExecutor(148): RpcExecutor default.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=5 2018-01-13 19:32:16,671 INFO [main] ipc.RpcExecutor(148): RpcExecutor priority.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=6 2018-01-13 19:32:16,671 INFO [main] ipc.RpcExecutor(148): RpcExecutor replication.FPBQ using fifo as call queue; numCallQueues=1; maxQueueLength=50; handlerCount=3 2018-01-13 19:32:16,671 INFO [main] ipc.RpcServerFactory(65): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2018-01-13 19:32:16,671 INFO [main] io.ByteBufferPool(83): Created ByteBufferPool with bufferSize : 65536 and maxPoolSize : 320 2018-01-13 19:32:16,674 INFO [main] ipc.NettyRpcServer(109): NettyRpcServer bind to address=/127.0.1.1:38841 2018-01-13 19:32:16,674 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=820320, freeSize=1117800608, maxSize=1118620928, heapSize=820320, minSize=1062689856, minFactor=0.95, multiSize=531344928, multiFactor=0.5, singleSize=265672464, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 19:32:16,674 INFO [main] hfile.CacheConfig(262): Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=820320, freeSize=1117800608, maxSize=1118620928, heapSize=820320, minSize=1062689856, minFactor=0.95, multiSize=531344928, multiFactor=0.5, singleSize=265672464, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 19:32:16,676 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 19:32:16,686 INFO [main] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 19:32:16,688 INFO [main] zookeeper.RecoverableZooKeeper(106): Process identifier=regionserver:38841 connecting to ZooKeeper ensemble=localhost:53209 2018-01-13 19:32:16,694 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:388410x0, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 19:32:16,695 DEBUG [main-EventThread] zookeeper.ZKWatcher(523): regionserver:38841-0x160ef4aa4c30003 connected 2018-01-13 19:32:16,696 DEBUG [main] zookeeper.ZKUtil(357): regionserver:38841-0x160ef4aa4c30003, quorum=localhost:53209, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 19:32:16,697 DEBUG [main] zookeeper.ZKUtil(357): regionserver:38841-0x160ef4aa4c30003, quorum=localhost:53209, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 19:32:16,697 DEBUG [main] ipc.RpcExecutor(254): Started 0 default.FPBQ.Fifo handlers, qsize=1 on port=38841 2018-01-13 19:32:16,704 DEBUG [main] ipc.RpcExecutor(254): Started 0 priority.FPBQ.Fifo handlers, qsize=1 on port=38841 2018-01-13 19:32:16,706 DEBUG [main] ipc.RpcExecutor(254): Started 0 replication.FPBQ.Fifo handlers, qsize=1 on port=38841 2018-01-13 19:32:16,708 INFO [M:0;ubuntu:36079] master.HMaster(1998): Adding backup master ZNode /hbase/backup-masters/ubuntu,36079,1515843135537 2018-01-13 19:32:16,719 DEBUG [M:0;ubuntu:36079] zookeeper.ZKUtil(355): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/ubuntu,36079,1515843135537 2018-01-13 19:32:16,745 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:40283-0x160ef4aa4c30002, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 19:32:16,745 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:44139-0x160ef4aa4c30001, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 19:32:16,746 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:38841-0x160ef4aa4c30003, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 19:32:16,745 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2018-01-13 19:32:16,749 DEBUG [M:0;ubuntu:36079] zookeeper.ZKUtil(355): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 19:32:16,751 INFO [M:0;ubuntu:36079] master.ActiveMasterManager(172): Deleting ZNode for /hbase/backup-masters/ubuntu,36079,1515843135537 from backup master directory 2018-01-13 19:32:16,765 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2018-01-13 19:32:16,766 WARN [M:0;ubuntu:36079] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 19:32:16,766 INFO [M:0;ubuntu:36079] master.ActiveMasterManager(181): Registered Active Master=ubuntu,36079,1515843135537 2018-01-13 19:32:16,765 DEBUG [main-EventThread] master.ActiveMasterManager(128): A master is now available 2018-01-13 19:32:16,768 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/ubuntu,36079,1515843135537 2018-01-13 19:32:16,774 INFO [M:0;ubuntu:36079] regionserver.ChunkCreator(408): Allocating MemStoreChunkPool with chunk size 2 MB, max count 533, initial count 0 2018-01-13 19:32:16,882 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39981 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-3a12d4a8-59eb-4588-98a1-23de755fb453:NORMAL:127.0.0.1:35477|RBW], ReplicaUC[[DISK]DS-2ae3c181-7033-4bb1-926b-79afb8ba9668:NORMAL:127.0.0.1:45841|RBW], ReplicaUC[[DISK]DS-2682b908-d71d-4edd-a7d6-c00e22c135f0:NORMAL:127.0.0.1:39981|FINALIZED]]} size 0 2018-01-13 19:32:16,883 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45841 is added to blk_1073741826_1002{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-3a12d4a8-59eb-4588-98a1-23de755fb453:NORMAL:127.0.0.1:35477|RBW], ReplicaUC[[DISK]DS-2ae3c181-7033-4bb1-926b-79afb8ba9668:NORMAL:127.0.0.1:45841|RBW], ReplicaUC[[DISK]DS-2682b908-d71d-4edd-a7d6-c00e22c135f0:NORMAL:127.0.0.1:39981|FINALIZED]]} size 0 2018-01-13 19:32:16,886 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35477 is added to blk_1073741826_1002 size 42 2018-01-13 19:32:16,889 DEBUG [M:0;ubuntu:36079] util.FSUtils(665): Created cluster ID file at hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/hbase.id with ID: dd9a41f6-7a7d-47f5-88fb-285f936c7138 2018-01-13 19:32:16,917 INFO [M:0;ubuntu:36079] master.MasterFileSystem(388): BOOTSTRAP: creating hbase:meta region 2018-01-13 19:32:16,921 INFO [M:0;ubuntu:36079] regionserver.HRegion(6772): creating HRegion hbase:meta HTD == 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, {NAME => 'info', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'false', BLOCKSIZE => '8192'}, {NAME => 'rep_barrier', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'rep_meta', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'rep_position', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'table', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433 Table name == hbase:meta 2018-01-13 19:32:16,971 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39981 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ae3c181-7033-4bb1-926b-79afb8ba9668:NORMAL:127.0.0.1:45841|RBW], ReplicaUC[[DISK]DS-3a12d4a8-59eb-4588-98a1-23de755fb453:NORMAL:127.0.0.1:35477|RBW], ReplicaUC[[DISK]DS-c786ffef-0cfe-40db-9609-e37ce51fcb98:NORMAL:127.0.0.1:39981|FINALIZED]]} size 0 2018-01-13 19:32:16,976 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45841 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-3a12d4a8-59eb-4588-98a1-23de755fb453:NORMAL:127.0.0.1:35477|RBW], ReplicaUC[[DISK]DS-c786ffef-0cfe-40db-9609-e37ce51fcb98:NORMAL:127.0.0.1:39981|FINALIZED], ReplicaUC[[DISK]DS-6e234590-c709-4272-a78c-d0a2418a7171:NORMAL:127.0.0.1:45841|FINALIZED]]} size 0 2018-01-13 19:32:16,976 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35477 is added to blk_1073741827_1003{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c786ffef-0cfe-40db-9609-e37ce51fcb98:NORMAL:127.0.0.1:39981|FINALIZED], ReplicaUC[[DISK]DS-6e234590-c709-4272-a78c-d0a2418a7171:NORMAL:127.0.0.1:45841|FINALIZED], ReplicaUC[[DISK]DS-7b74f716-787f-414f-bbbf-7d604b248220:NORMAL:127.0.0.1:35477|FINALIZED]]} size 0 2018-01-13 19:32:16,981 DEBUG [M:0;ubuntu:36079] regionserver.HRegion(807): Instantiated hbase:meta,,1.1588230740 2018-01-13 19:32:17,024 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/info 2018-01-13 19:32:17,036 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 19:32:17,039 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=820320, freeSize=1117800608, maxSize=1118620928, heapSize=820320, minSize=1062689856, minFactor=0.95, multiSize=531344928, multiFactor=0.5, singleSize=265672464, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 19:32:17,049 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 19:32:17,087 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/rep_barrier 2018-01-13 19:32:17,087 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 19:32:17,088 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=820320, freeSize=1117800608, maxSize=1118620928, heapSize=820320, minSize=1062689856, minFactor=0.95, multiSize=531344928, multiFactor=0.5, singleSize=265672464, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 19:32:17,089 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 19:32:17,096 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/rep_meta 2018-01-13 19:32:17,096 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 19:32:17,097 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_meta: blockCache=LruBlockCache{blockCount=0, currentSize=820320, freeSize=1117800608, maxSize=1118620928, heapSize=820320, minSize=1062689856, minFactor=0.95, multiSize=531344928, multiFactor=0.5, singleSize=265672464, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 19:32:17,097 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 19:32:17,102 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/rep_position 2018-01-13 19:32:17,103 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 19:32:17,103 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_position: blockCache=LruBlockCache{blockCount=0, currentSize=820320, freeSize=1117800608, maxSize=1118620928, heapSize=820320, minSize=1062689856, minFactor=0.95, multiSize=531344928, multiFactor=0.5, singleSize=265672464, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 19:32:17,104 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 19:32:17,110 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/table 2018-01-13 19:32:17,110 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 19:32:17,110 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=820320, freeSize=1117800608, maxSize=1118620928, heapSize=820320, minSize=1062689856, minFactor=0.95, multiSize=531344928, multiFactor=0.5, singleSize=265672464, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 19:32:17,111 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 19:32:17,123 DEBUG [M:0;ubuntu:36079] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740 2018-01-13 19:32:17,141 DEBUG [M:0;ubuntu:36079] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushSize/# of families (25.6M)) instead. 2018-01-13 19:32:17,153 DEBUG [M:0;ubuntu:36079] wal.WALSplitter(675): Wrote file=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 19:32:17,153 INFO [M:0;ubuntu:36079] regionserver.HRegion(960): Onlined 1588230740; next sequenceid=2 2018-01-13 19:32:17,154 DEBUG [M:0;ubuntu:36079] regionserver.HRegion(1481): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2018-01-13 19:32:17,154 DEBUG [M:0;ubuntu:36079] regionserver.HRegion(1520): Updates disabled for region hbase:meta,,1.1588230740 2018-01-13 19:32:17,155 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed info 2018-01-13 19:32:17,158 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_barrier 2018-01-13 19:32:17,158 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_meta 2018-01-13 19:32:17,158 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_position 2018-01-13 19:32:17,158 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed table 2018-01-13 19:32:17,160 INFO [M:0;ubuntu:36079] regionserver.HRegion(1633): Closed hbase:meta,,1.1588230740 2018-01-13 19:32:17,202 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45841 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2682b908-d71d-4edd-a7d6-c00e22c135f0:NORMAL:127.0.0.1:39981|RBW], ReplicaUC[[DISK]DS-7b74f716-787f-414f-bbbf-7d604b248220:NORMAL:127.0.0.1:35477|RBW], ReplicaUC[[DISK]DS-2ae3c181-7033-4bb1-926b-79afb8ba9668:NORMAL:127.0.0.1:45841|RBW]]} size 0 2018-01-13 19:32:17,206 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35477 is added to blk_1073741828_1004{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2682b908-d71d-4edd-a7d6-c00e22c135f0:NORMAL:127.0.0.1:39981|RBW], ReplicaUC[[DISK]DS-2ae3c181-7033-4bb1-926b-79afb8ba9668:NORMAL:127.0.0.1:45841|RBW], ReplicaUC[[DISK]DS-3a12d4a8-59eb-4588-98a1-23de755fb453:NORMAL:127.0.0.1:35477|FINALIZED]]} size 0 2018-01-13 19:32:17,206 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39981 is added to blk_1073741828_1004 size 2318 2018-01-13 19:32:17,211 DEBUG [M:0;ubuntu:36079] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2018-01-13 19:32:17,241 INFO [M:0;ubuntu:36079] fs.HFileSystem(349): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2018-01-13 19:32:17,257 INFO [M:0;ubuntu:36079] coordination.ZKSplitLogManagerCoordination(492): Found 0 orphan tasks and 0 rescan nodes 2018-01-13 19:32:17,258 DEBUG [M:0;ubuntu:36079] util.FSTableDescriptors(292): Fetching table descriptors from the filesystem. 2018-01-13 19:32:17,284 INFO [M:0;ubuntu:36079] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x4327cbfb to localhost:53209, session timeout 90000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 19:32:17,327 DEBUG [M:0;ubuntu:36079] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23c34408, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 19:32:17,355 INFO [M:0;ubuntu:36079] balancer.BaseLoadBalancer(1037): slop=0.001, tablesOnMaster=false, systemTablesOnMaster=false 2018-01-13 19:32:17,359 INFO [M:0;ubuntu:36079] balancer.StochasticLoadBalancer(214): Loaded config; maxSteps=1000000, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, etc. 2018-01-13 19:32:17,363 DEBUG [M:0;ubuntu:36079] zookeeper.ZKUtil(357): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2018-01-13 19:32:17,365 DEBUG [M:0;ubuntu:36079] zookeeper.ZKUtil(357): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2018-01-13 19:32:17,371 DEBUG [M:0;ubuntu:36079] zookeeper.ZKUtil(357): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2018-01-13 19:32:17,373 DEBUG [M:0;ubuntu:36079] zookeeper.ZKUtil(357): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2018-01-13 19:32:17,394 INFO [M:0;ubuntu:36079] assignment.AssignmentManager(219): Starting assignment manager 2018-01-13 19:32:17,411 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:40283-0x160ef4aa4c30002, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 19:32:17,411 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:44139-0x160ef4aa4c30001, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 19:32:17,411 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 19:32:17,411 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:38841-0x160ef4aa4c30003, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2018-01-13 19:32:17,412 INFO [M:0;ubuntu:36079] master.HMaster(746): Server active/primary master=ubuntu,36079,1515843135537, sessionid=0x160ef4aa4c30000, setting cluster-up flag (Was=false) 2018-01-13 19:32:17,446 INFO [M:0;ubuntu:36079] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2018-01-13 19:32:17,448 DEBUG [M:0;ubuntu:36079] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:ubuntu,36079,1515843135537 2018-01-13 19:32:17,456 INFO [M:0;ubuntu:36079] procedure.ZKProcedureUtil(271): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2018-01-13 19:32:17,458 DEBUG [M:0;ubuntu:36079] procedure.ZKProcedureCoordinator(248): Starting the controller for procedure member:ubuntu,36079,1515843135537 2018-01-13 19:32:17,491 INFO [M:0;ubuntu:36079] master.MasterCoprocessorHost(152): System coprocessor loading is enabled 2018-01-13 19:32:17,493 DEBUG [M:0;ubuntu:36079] executor.ExecutorService(92): Starting executor service name=MASTER_OPEN_REGION-ubuntu:36079, corePoolSize=5, maxPoolSize=5 2018-01-13 19:32:17,494 DEBUG [M:0;ubuntu:36079] executor.ExecutorService(92): Starting executor service name=MASTER_CLOSE_REGION-ubuntu:36079, corePoolSize=5, maxPoolSize=5 2018-01-13 19:32:17,494 DEBUG [M:0;ubuntu:36079] executor.ExecutorService(92): Starting executor service name=MASTER_SERVER_OPERATIONS-ubuntu:36079, corePoolSize=5, maxPoolSize=5 2018-01-13 19:32:17,494 DEBUG [M:0;ubuntu:36079] executor.ExecutorService(92): Starting executor service name=MASTER_META_SERVER_OPERATIONS-ubuntu:36079, corePoolSize=5, maxPoolSize=5 2018-01-13 19:32:17,494 DEBUG [M:0;ubuntu:36079] executor.ExecutorService(92): Starting executor service name=M_LOG_REPLAY_OPS-ubuntu:36079, corePoolSize=10, maxPoolSize=10 2018-01-13 19:32:17,494 DEBUG [M:0;ubuntu:36079] executor.ExecutorService(92): Starting executor service name=MASTER_TABLE_OPERATIONS-ubuntu:36079, corePoolSize=1, maxPoolSize=1 2018-01-13 19:32:17,530 INFO [M:0;ubuntu:36079] procedure2.ProcedureExecutor(513): Starting ProcedureExecutor Worker threads (ProcExecWrkr)=16 2018-01-13 19:32:17,521 INFO [RS:2;ubuntu:38841] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x14f00d46 to localhost:53209, session timeout 90000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 19:32:17,531 INFO [RS:0;ubuntu:44139] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x6f8ca84a to localhost:53209, session timeout 90000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 19:32:17,531 INFO [RS:1;ubuntu:40283] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x4e704e39 to localhost:53209, session timeout 90000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 19:32:17,536 INFO [M:0;ubuntu:36079] wal.WALProcedureStore(361): Starting WAL Procedure Store lease recovery 2018-01-13 19:32:17,542 DEBUG [RS:1;ubuntu:40283] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@40b7b987, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 19:32:17,544 INFO [RS:1;ubuntu:40283] regionserver.HRegionServer(866): ClusterId : dd9a41f6-7a7d-47f5-88fb-285f936c7138 2018-01-13 19:32:17,546 DEBUG [RS:0;ubuntu:44139] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@614b5bcb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 19:32:17,547 INFO [RS:0;ubuntu:44139] regionserver.HRegionServer(866): ClusterId : dd9a41f6-7a7d-47f5-88fb-285f936c7138 2018-01-13 19:32:17,549 DEBUG [RS:1;ubuntu:40283] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2018-01-13 19:32:17,549 DEBUG [RS:0;ubuntu:44139] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2018-01-13 19:32:17,554 DEBUG [RS:2;ubuntu:38841] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a37e1a8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 19:32:17,554 DEBUG [RS:1;ubuntu:40283] zookeeper.RecoverableZooKeeper(571): Node /hbase/flush-table-proc/acquired already exists 2018-01-13 19:32:17,555 DEBUG [RS:0;ubuntu:44139] zookeeper.RecoverableZooKeeper(571): Node /hbase/flush-table-proc/acquired already exists 2018-01-13 19:32:17,556 INFO [RS:2;ubuntu:38841] regionserver.HRegionServer(866): ClusterId : dd9a41f6-7a7d-47f5-88fb-285f936c7138 2018-01-13 19:32:17,556 DEBUG [RS:2;ubuntu:38841] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2018-01-13 19:32:17,557 DEBUG [M:0;ubuntu:36079] wal.WALProcedureStore(1071): Roll new state log: 1 2018-01-13 19:32:17,557 DEBUG [RS:1;ubuntu:40283] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2018-01-13 19:32:17,557 DEBUG [RS:1;ubuntu:40283] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2018-01-13 19:32:17,557 DEBUG [RS:0;ubuntu:44139] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2018-01-13 19:32:17,557 DEBUG [RS:0;ubuntu:44139] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2018-01-13 19:32:17,559 INFO [M:0;ubuntu:36079] wal.WALProcedureStore(390): Lease acquired for flushLogId: 1 2018-01-13 19:32:17,559 DEBUG [RS:1;ubuntu:40283] zookeeper.RecoverableZooKeeper(571): Node /hbase/online-snapshot/acquired already exists 2018-01-13 19:32:17,561 DEBUG [RS:1;ubuntu:40283] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2018-01-13 19:32:17,561 INFO [M:0;ubuntu:36079] procedure2.ProcedureExecutor(534): Recover store (WALProcedureStore) lease: 23msec 2018-01-13 19:32:17,562 DEBUG [RS:1;ubuntu:40283] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44882549, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=ubuntu/127.0.1.1:0 2018-01-13 19:32:17,562 DEBUG [M:0;ubuntu:36079] wal.WALProcedureStore(409): No state logs to replay. 2018-01-13 19:32:17,563 DEBUG [M:0;ubuntu:36079] procedure2.ProcedureExecutor$1(317): Load max pid=0 2018-01-13 19:32:17,563 INFO [M:0;ubuntu:36079] procedure2.ProcedureExecutor(548): Load store (WALProcedureStore): 2msec 2018-01-13 19:32:17,564 DEBUG [RS:0;ubuntu:44139] zookeeper.RecoverableZooKeeper(571): Node /hbase/online-snapshot/acquired already exists 2018-01-13 19:32:17,564 DEBUG [RS:2;ubuntu:38841] zookeeper.RecoverableZooKeeper(571): Node /hbase/flush-table-proc/acquired already exists 2018-01-13 19:32:17,565 DEBUG [RS:1;ubuntu:40283] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:1;ubuntu:40283 2018-01-13 19:32:17,566 DEBUG [RS:2;ubuntu:38841] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2018-01-13 19:32:17,567 DEBUG [RS:2;ubuntu:38841] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2018-01-13 19:32:17,570 INFO [RS:1;ubuntu:40283] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2018-01-13 19:32:17,570 INFO [RS:1;ubuntu:40283] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2018-01-13 19:32:17,574 DEBUG [RS:2;ubuntu:38841] zookeeper.RecoverableZooKeeper(571): Node /hbase/online-snapshot/acquired already exists 2018-01-13 19:32:17,574 DEBUG [RS:0;ubuntu:44139] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2018-01-13 19:32:17,574 DEBUG [RS:0;ubuntu:44139] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6574c33a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=ubuntu/127.0.1.1:0 2018-01-13 19:32:17,575 DEBUG [RS:2;ubuntu:38841] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2018-01-13 19:32:17,576 DEBUG [RS:2;ubuntu:38841] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e086636, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=ubuntu/127.0.1.1:0 2018-01-13 19:32:17,576 INFO [RS:1;ubuntu:40283] regionserver.HRegionServer(2612): reportForDuty to master=ubuntu,36079,1515843135537 with port=40283, startcode=1515843136612 2018-01-13 19:32:17,587 DEBUG [RS:0;ubuntu:44139] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:0;ubuntu:44139 2018-01-13 19:32:17,588 INFO [M:0;ubuntu:36079] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1515843167588 2018-01-13 19:32:17,589 DEBUG [RS:2;ubuntu:38841] regionserver.ShutdownHook(88): Installed shutdown hook thread: Shutdownhook:RS:2;ubuntu:38841 2018-01-13 19:32:17,589 INFO [RS:2;ubuntu:38841] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2018-01-13 19:32:17,589 INFO [RS:2;ubuntu:38841] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2018-01-13 19:32:17,588 INFO [RS:0;ubuntu:44139] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2018-01-13 19:32:17,590 INFO [RS:0;ubuntu:44139] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2018-01-13 19:32:17,590 INFO [M:0;ubuntu:36079] procedure2.RemoteProcedureDispatcher(97): Starting procedure remote dispatcher; threads=128, queueMaxSize=32, operationDelay=150 2018-01-13 19:32:17,593 INFO [RS:0;ubuntu:44139] regionserver.HRegionServer(2612): reportForDuty to master=ubuntu,36079,1515843135537 with port=44139, startcode=1515843136509 2018-01-13 19:32:17,593 INFO [RS:2;ubuntu:38841] regionserver.HRegionServer(2612): reportForDuty to master=ubuntu,36079,1515843135537 with port=38841, startcode=1515843136669 2018-01-13 19:32:17,596 DEBUG [M:0;ubuntu:36079] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2018-01-13 19:32:17,597 INFO [M:0;ubuntu:36079] zookeeper.RecoverableZooKeeper(106): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:53209 2018-01-13 19:32:17,597 DEBUG [M:0;ubuntu:36079] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2018-01-13 19:32:17,599 DEBUG [M:0;ubuntu:36079] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2018-01-13 19:32:17,599 INFO [M:0;ubuntu:36079] cleaner.CleanerChore(113): Cleaner pool size is 2 2018-01-13 19:32:17,599 INFO [M:0;ubuntu:36079] cleaner.LogCleaner(122): Creating OldWALs cleaners with size: 2 2018-01-13 19:32:17,604 DEBUG [M:0;ubuntu:36079-EventThread] zookeeper.ZKWatcher(460): replicationLogCleaner0x0, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2018-01-13 19:32:17,608 DEBUG [M:0;ubuntu:36079-EventThread] zookeeper.ZKWatcher(523): replicationLogCleaner-0x160ef4aa4c30008 connected 2018-01-13 19:32:17,609 DEBUG [M:0;ubuntu:36079] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2018-01-13 19:32:17,613 DEBUG [M:0;ubuntu:36079] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2018-01-13 19:32:17,614 DEBUG [M:0;ubuntu:36079] cleaner.CleanerChore(166): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2018-01-13 19:32:17,615 DEBUG [M:0;ubuntu:36079] cleaner.HFileCleaner(207): Starting hfile cleaner for large files: Thread[M:0;ubuntu:36079-HFileCleaner.large.0-1515843137615,5,main] 2018-01-13 19:32:17,616 DEBUG [M:0;ubuntu:36079] cleaner.HFileCleaner(222): Starting hfile cleaner for small files: Thread[M:0;ubuntu:36079-HFileCleaner.small.0-1515843137615,5,main] 2018-01-13 19:32:17,626 DEBUG [RS:2;ubuntu:38841] ipc.NettyRpcConnection(255): Connecting to ubuntu/127.0.1.1:36079 2018-01-13 19:32:17,629 INFO [M:0;ubuntu:36079] master.HMaster(850): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1515843136741, completionTime=-1 2018-01-13 19:32:17,629 INFO [M:0;ubuntu:36079] master.ServerManager(871): Waiting on RegionServer count=0 to settle; waited=0ms, expecting min=3 server(s), max=3 server(s), timeout=4500ms, lastChange=0ms 2018-01-13 19:32:17,632 DEBUG [RS:0;ubuntu:44139] ipc.NettyRpcConnection(255): Connecting to ubuntu/127.0.1.1:36079 2018-01-13 19:32:17,626 DEBUG [RS:1;ubuntu:40283] ipc.NettyRpcConnection(255): Connecting to ubuntu/127.0.1.1:36079 2018-01-13 19:32:17,704 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(527): Auth successful for zhangduo.hfs.0 (auth:SIMPLE) 2018-01-13 19:32:17,706 INFO [RS-EventLoopGroup-1-4] ipc.ServerRpcConnection(527): Auth successful for zhangduo.hfs.2 (auth:SIMPLE) 2018-01-13 19:32:17,704 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(527): Auth successful for zhangduo.hfs.1 (auth:SIMPLE) 2018-01-13 19:32:17,711 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(557): Connection from 127.0.1.1 port: 40907 with version info: version: "2.0.0-beta-1" url: "git://ubuntu/home/zhangduo/hbase/code" revision: "a90cf1bc134a14ba7b692c8eb6ee7cddc4b857bf" user: "zhangduo" date: "Fri Jan 12 21:37:44 CST 2018" src_checksum: "5c5be6fe364dffb36035e82ded054eaa" version_major: 2 version_minor: 0 2018-01-13 19:32:17,712 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(557): Connection from 127.0.1.1 port: 48353 with version info: version: "2.0.0-beta-1" url: "git://ubuntu/home/zhangduo/hbase/code" revision: "a90cf1bc134a14ba7b692c8eb6ee7cddc4b857bf" user: "zhangduo" date: "Fri Jan 12 21:37:44 CST 2018" src_checksum: "5c5be6fe364dffb36035e82ded054eaa" version_major: 2 version_minor: 0 2018-01-13 19:32:17,714 INFO [RS-EventLoopGroup-1-4] ipc.ServerRpcConnection(557): Connection from 127.0.1.1 port: 50195 with version info: version: "2.0.0-beta-1" url: "git://ubuntu/home/zhangduo/hbase/code" revision: "a90cf1bc134a14ba7b692c8eb6ee7cddc4b857bf" user: "zhangduo" date: "Fri Jan 12 21:37:44 CST 2018" src_checksum: "5c5be6fe364dffb36035e82ded054eaa" version_major: 2 version_minor: 0 2018-01-13 19:32:17,746 INFO [RpcServer.default.FPBQ.Fifo.handler=4,queue=0,port=36079] master.ServerManager(432): Registering server=ubuntu,40283,1515843136612 2018-01-13 19:32:17,747 INFO [RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=36079] master.ServerManager(432): Registering server=ubuntu,44139,1515843136509 2018-01-13 19:32:17,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36079] master.ServerManager(432): Registering server=ubuntu,38841,1515843136669 2018-01-13 19:32:17,766 DEBUG [RS:0;ubuntu:44139] regionserver.HRegionServer(1494): Config from master: hbase.rootdir=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433 2018-01-13 19:32:17,766 DEBUG [RS:2;ubuntu:38841] regionserver.HRegionServer(1494): Config from master: hbase.rootdir=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433 2018-01-13 19:32:17,767 DEBUG [RS:0;ubuntu:44139] regionserver.HRegionServer(1494): Config from master: fs.defaultFS=hdfs://localhost:44545 2018-01-13 19:32:17,767 DEBUG [RS:0;ubuntu:44139] regionserver.HRegionServer(1494): Config from master: hbase.master.info.port=-1 2018-01-13 19:32:17,766 DEBUG [RS:1;ubuntu:40283] regionserver.HRegionServer(1494): Config from master: hbase.rootdir=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433 2018-01-13 19:32:17,767 DEBUG [RS:2;ubuntu:38841] regionserver.HRegionServer(1494): Config from master: fs.defaultFS=hdfs://localhost:44545 2018-01-13 19:32:17,770 DEBUG [RS:2;ubuntu:38841] regionserver.HRegionServer(1494): Config from master: hbase.master.info.port=-1 2018-01-13 19:32:17,771 DEBUG [RS:1;ubuntu:40283] regionserver.HRegionServer(1494): Config from master: fs.defaultFS=hdfs://localhost:44545 2018-01-13 19:32:17,771 DEBUG [RS:1;ubuntu:40283] regionserver.HRegionServer(1494): Config from master: hbase.master.info.port=-1 2018-01-13 19:32:17,780 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 19:32:17,781 DEBUG [RS:2;ubuntu:38841] zookeeper.ZKUtil(355): regionserver:38841-0x160ef4aa4c30003, quorum=localhost:53209, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,38841,1515843136669 2018-01-13 19:32:17,782 INFO [M:0;ubuntu:36079] master.ServerManager(891): Finished wait on RegionServer count=3; waited=153ms, expected min=3 server(s), max=3 server(s), master is running 2018-01-13 19:32:17,782 WARN [RS:2;ubuntu:38841] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 19:32:17,786 DEBUG [RS:2;ubuntu:38841] regionserver.HRegionServer(1800): logDir=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/WALs/ubuntu,38841,1515843136669 2018-01-13 19:32:17,788 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,38841,1515843136669 2018-01-13 19:32:17,788 DEBUG [RS:1;ubuntu:40283] zookeeper.ZKUtil(355): regionserver:40283-0x160ef4aa4c30002, quorum=localhost:53209, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,40283,1515843136612 2018-01-13 19:32:17,789 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,40283,1515843136612 2018-01-13 19:32:17,789 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,44139,1515843136509 2018-01-13 19:32:17,789 DEBUG [RS:0;ubuntu:44139] zookeeper.ZKUtil(355): regionserver:44139-0x160ef4aa4c30001, quorum=localhost:53209, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,44139,1515843136509 2018-01-13 19:32:17,789 WARN [RS:0;ubuntu:44139] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 19:32:17,795 DEBUG [RS:0;ubuntu:44139] regionserver.HRegionServer(1800): logDir=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/WALs/ubuntu,44139,1515843136509 2018-01-13 19:32:17,788 WARN [RS:1;ubuntu:40283] hbase.ZNodeClearer(62): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2018-01-13 19:32:17,796 DEBUG [RS:1;ubuntu:40283] regionserver.HRegionServer(1800): logDir=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/WALs/ubuntu,40283,1515843136612 2018-01-13 19:32:17,813 INFO [M:0;ubuntu:36079] master.HMaster(3492): Running RecoverMetaProcedure to ensure proper hbase:meta deploy. 2018-01-13 19:32:17,821 INFO [RS:0;ubuntu:44139] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 19:32:17,821 INFO [RS:2;ubuntu:38841] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 19:32:17,821 INFO [RS:1;ubuntu:40283] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 19:32:17,832 DEBUG [RS:1;ubuntu:40283] zookeeper.ZKUtil(355): regionserver:40283-0x160ef4aa4c30002, quorum=localhost:53209, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,38841,1515843136669 2018-01-13 19:32:17,833 DEBUG [RS:1;ubuntu:40283] zookeeper.ZKUtil(355): regionserver:40283-0x160ef4aa4c30002, quorum=localhost:53209, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,40283,1515843136612 2018-01-13 19:32:17,833 DEBUG [RS:1;ubuntu:40283] zookeeper.ZKUtil(355): regionserver:40283-0x160ef4aa4c30002, quorum=localhost:53209, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,44139,1515843136509 2018-01-13 19:32:17,836 DEBUG [RS:0;ubuntu:44139] zookeeper.ZKUtil(355): regionserver:44139-0x160ef4aa4c30001, quorum=localhost:53209, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,38841,1515843136669 2018-01-13 19:32:17,837 DEBUG [RS:0;ubuntu:44139] zookeeper.ZKUtil(355): regionserver:44139-0x160ef4aa4c30001, quorum=localhost:53209, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,40283,1515843136612 2018-01-13 19:32:17,839 DEBUG [RS:2;ubuntu:38841] zookeeper.ZKUtil(355): regionserver:38841-0x160ef4aa4c30003, quorum=localhost:53209, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,38841,1515843136669 2018-01-13 19:32:17,839 DEBUG [RS:0;ubuntu:44139] zookeeper.ZKUtil(355): regionserver:44139-0x160ef4aa4c30001, quorum=localhost:53209, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,44139,1515843136509 2018-01-13 19:32:17,840 DEBUG [RS:2;ubuntu:38841] zookeeper.ZKUtil(355): regionserver:38841-0x160ef4aa4c30003, quorum=localhost:53209, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,40283,1515843136612 2018-01-13 19:32:17,842 DEBUG [RS:2;ubuntu:38841] zookeeper.ZKUtil(355): regionserver:38841-0x160ef4aa4c30003, quorum=localhost:53209, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,44139,1515843136509 2018-01-13 19:32:17,845 INFO [RS:1;ubuntu:40283] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x52e7e3a1 to localhost:53209, session timeout 90000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 19:32:17,848 INFO [RS:0;ubuntu:44139] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x79489deb to localhost:53209, session timeout 90000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 19:32:17,847 INFO [RS:2;ubuntu:38841] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x3c16b72e to localhost:53209, session timeout 90000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 19:32:17,889 DEBUG [RS:1;ubuntu:40283] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@75f95cd6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 19:32:17,889 DEBUG [RS:1;ubuntu:40283] regionserver.Replication(149): ReplicationStatisticsThread 300 2018-01-13 19:32:17,890 DEBUG [RS:2;ubuntu:38841] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@19efef9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 19:32:17,890 DEBUG [RS:2;ubuntu:38841] regionserver.Replication(149): ReplicationStatisticsThread 300 2018-01-13 19:32:17,898 DEBUG [RS:0;ubuntu:44139] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@fa4e870, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 19:32:17,899 DEBUG [RS:0;ubuntu:44139] regionserver.Replication(149): ReplicationStatisticsThread 300 2018-01-13 19:32:17,900 INFO [RS:2;ubuntu:38841] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-01-13 19:32:17,900 INFO [RS:0;ubuntu:44139] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-01-13 19:32:17,900 INFO [RS:1;ubuntu:40283] regionserver.MetricsRegionServerWrapperImpl(144): Computing regionserver metrics every 5000 milliseconds 2018-01-13 19:32:17,939 INFO [RS:0;ubuntu:44139] regionserver.MemStoreFlusher(105): globalMemStoreLimit=1.0 G, globalMemStoreLimitLowMark=1013.5 M, Offheap=false 2018-01-13 19:32:17,941 INFO [RS:1;ubuntu:40283] regionserver.MemStoreFlusher(105): globalMemStoreLimit=1.0 G, globalMemStoreLimitLowMark=1013.5 M, Offheap=false 2018-01-13 19:32:17,948 INFO [RS:2;ubuntu:38841] regionserver.MemStoreFlusher(105): globalMemStoreLimit=1.0 G, globalMemStoreLimitLowMark=1013.5 M, Offheap=false 2018-01-13 19:32:17,952 INFO [RS:1;ubuntu:40283] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2018-01-13 19:32:17,953 INFO [RS:1;ubuntu:40283] regionserver.HRegionServer$CompactionChecker(1694): CompactionChecker runs every 1sec 2018-01-13 19:32:17,952 INFO [RS:2;ubuntu:38841] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2018-01-13 19:32:17,952 INFO [RS:0;ubuntu:44139] throttle.PressureAwareCompactionThroughputController(134): Compaction throughput configurations, higher bound: 20.00 MB/sec, lower bound 10.00 MB/sec, off peak: unlimited, tuning period: 60000 ms 2018-01-13 19:32:17,959 INFO [RS:2;ubuntu:38841] regionserver.HRegionServer$CompactionChecker(1694): CompactionChecker runs every 1sec 2018-01-13 19:32:17,960 INFO [RS:0;ubuntu:44139] regionserver.HRegionServer$CompactionChecker(1694): CompactionChecker runs every 1sec 2018-01-13 19:32:17,982 DEBUG [RS:2;ubuntu:38841] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-ubuntu:38841, corePoolSize=3, maxPoolSize=3 2018-01-13 19:32:17,982 DEBUG [RS:2;ubuntu:38841] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-ubuntu:38841, corePoolSize=1, maxPoolSize=1 2018-01-13 19:32:17,982 DEBUG [RS:2;ubuntu:38841] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-ubuntu:38841, corePoolSize=3, maxPoolSize=3 2018-01-13 19:32:17,982 DEBUG [RS:2;ubuntu:38841] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-ubuntu:38841, corePoolSize=3, maxPoolSize=3 2018-01-13 19:32:17,982 DEBUG [RS:2;ubuntu:38841] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-ubuntu:38841, corePoolSize=1, maxPoolSize=1 2018-01-13 19:32:17,982 DEBUG [RS:2;ubuntu:38841] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-ubuntu:38841, corePoolSize=2, maxPoolSize=2 2018-01-13 19:32:17,983 DEBUG [RS:2;ubuntu:38841] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-ubuntu:38841, corePoolSize=10, maxPoolSize=10 2018-01-13 19:32:17,983 DEBUG [RS:2;ubuntu:38841] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-ubuntu:38841, corePoolSize=3, maxPoolSize=3 2018-01-13 19:32:17,983 DEBUG [RS:2;ubuntu:38841] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-ubuntu:38841, corePoolSize=2, maxPoolSize=2 2018-01-13 19:32:17,982 DEBUG [RS:0;ubuntu:44139] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-ubuntu:44139, corePoolSize=3, maxPoolSize=3 2018-01-13 19:32:17,982 DEBUG [RS:1;ubuntu:40283] executor.ExecutorService(92): Starting executor service name=RS_OPEN_REGION-ubuntu:40283, corePoolSize=3, maxPoolSize=3 2018-01-13 19:32:17,986 DEBUG [RS:0;ubuntu:44139] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-ubuntu:44139, corePoolSize=1, maxPoolSize=1 2018-01-13 19:32:17,986 DEBUG [RS:1;ubuntu:40283] executor.ExecutorService(92): Starting executor service name=RS_OPEN_META-ubuntu:40283, corePoolSize=1, maxPoolSize=1 2018-01-13 19:32:17,986 DEBUG [RS:1;ubuntu:40283] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-ubuntu:40283, corePoolSize=3, maxPoolSize=3 2018-01-13 19:32:17,986 DEBUG [RS:1;ubuntu:40283] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-ubuntu:40283, corePoolSize=3, maxPoolSize=3 2018-01-13 19:32:17,986 DEBUG [RS:1;ubuntu:40283] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-ubuntu:40283, corePoolSize=1, maxPoolSize=1 2018-01-13 19:32:17,986 DEBUG [RS:1;ubuntu:40283] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-ubuntu:40283, corePoolSize=2, maxPoolSize=2 2018-01-13 19:32:17,987 DEBUG [RS:1;ubuntu:40283] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-ubuntu:40283, corePoolSize=10, maxPoolSize=10 2018-01-13 19:32:17,987 DEBUG [RS:1;ubuntu:40283] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-ubuntu:40283, corePoolSize=3, maxPoolSize=3 2018-01-13 19:32:17,987 DEBUG [RS:1;ubuntu:40283] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-ubuntu:40283, corePoolSize=2, maxPoolSize=2 2018-01-13 19:32:17,986 DEBUG [RS:0;ubuntu:44139] executor.ExecutorService(92): Starting executor service name=RS_OPEN_PRIORITY_REGION-ubuntu:44139, corePoolSize=3, maxPoolSize=3 2018-01-13 19:32:17,987 DEBUG [RS:0;ubuntu:44139] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_REGION-ubuntu:44139, corePoolSize=3, maxPoolSize=3 2018-01-13 19:32:17,987 DEBUG [RS:0;ubuntu:44139] executor.ExecutorService(92): Starting executor service name=RS_CLOSE_META-ubuntu:44139, corePoolSize=1, maxPoolSize=1 2018-01-13 19:32:17,987 DEBUG [RS:0;ubuntu:44139] executor.ExecutorService(92): Starting executor service name=RS_LOG_REPLAY_OPS-ubuntu:44139, corePoolSize=2, maxPoolSize=2 2018-01-13 19:32:17,987 DEBUG [RS:0;ubuntu:44139] executor.ExecutorService(92): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-ubuntu:44139, corePoolSize=10, maxPoolSize=10 2018-01-13 19:32:17,987 DEBUG [RS:0;ubuntu:44139] executor.ExecutorService(92): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-ubuntu:44139, corePoolSize=3, maxPoolSize=3 2018-01-13 19:32:17,988 DEBUG [RS:0;ubuntu:44139] executor.ExecutorService(92): Starting executor service name=RS_REFRESH_PEER-ubuntu:44139, corePoolSize=2, maxPoolSize=2 2018-01-13 19:32:18,037 INFO [RS:2;ubuntu:38841] regionserver.HeapMemoryManager(210): Starting HeapMemoryTuner chore. 2018-01-13 19:32:18,039 INFO [SplitLogWorker-ubuntu:38841] regionserver.SplitLogWorker(130): SplitLogWorker ubuntu,38841,1515843136669 starting 2018-01-13 19:32:18,043 DEBUG [M:0;ubuntu:36079] procedure2.ProcedureExecutor(869): Stored pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 19:32:18,058 INFO [RS:0;ubuntu:44139] regionserver.HeapMemoryManager(210): Starting HeapMemoryTuner chore. 2018-01-13 19:32:18,065 INFO [RS:1;ubuntu:40283] regionserver.HeapMemoryManager(210): Starting HeapMemoryTuner chore. 2018-01-13 19:32:18,069 INFO [SplitLogWorker-ubuntu:44139] regionserver.SplitLogWorker(130): SplitLogWorker ubuntu,44139,1515843136509 starting 2018-01-13 19:32:18,072 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 19:32:18,072 INFO [ProcExecWrkr-1] procedure.RecoverMetaProcedure(106): Start pid=1, state=RUNNABLE:RECOVER_META_SPLIT_LOGS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 19:32:18,074 DEBUG [ProcExecWrkr-1] zookeeper.ZKUtil(614): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-01-13 19:32:18,066 INFO [SplitLogWorker-ubuntu:40283] regionserver.SplitLogWorker(130): SplitLogWorker ubuntu,40283,1515843136612 starting 2018-01-13 19:32:18,080 DEBUG [ProcExecWrkr-1] master.MasterWalManager(169): No log files to split, proceeding... 2018-01-13 19:32:18,092 DEBUG [ProcExecWrkr-1] procedure2.StateMachineProcedure(180): pid=1, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true 2018-01-13 19:32:18,093 DEBUG [ProcExecWrkr-1] zookeeper.ZKUtil(614): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2018-01-13 19:32:18,093 INFO [ProcExecWrkr-1] procedure.RecoverMetaProcedure(137): pid=1, state=RUNNABLE:RECOVER_META_ASSIGN_REGIONS; RecoverMetaProcedure failedMetaServer=null, splitWal=true; Retaining meta assignment to server=null 2018-01-13 19:32:18,097 INFO [ProcExecWrkr-1] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740}] 2018-01-13 19:32:18,121 INFO [ProcExecWrkr-2] procedure.MasterProcedureScheduler(883): pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740 hbase:meta hbase:meta,,1.1588230740 2018-01-13 19:32:18,121 INFO [RS:2;ubuntu:38841] regionserver.HRegionServer(1535): Serving as ubuntu,38841,1515843136669, RpcServer on ubuntu/127.0.1.1:38841, sessionid=0x160ef4aa4c30003 2018-01-13 19:32:18,122 INFO [RS:0;ubuntu:44139] regionserver.HRegionServer(1535): Serving as ubuntu,44139,1515843136509, RpcServer on ubuntu/127.0.1.1:44139, sessionid=0x160ef4aa4c30001 2018-01-13 19:32:18,122 DEBUG [RS:0;ubuntu:44139] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2018-01-13 19:32:18,122 DEBUG [RS:0;ubuntu:44139] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager ubuntu,44139,1515843136509 2018-01-13 19:32:18,122 DEBUG [RS:0;ubuntu:44139] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'ubuntu,44139,1515843136509' 2018-01-13 19:32:18,123 DEBUG [RS:0;ubuntu:44139] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-01-13 19:32:18,124 DEBUG [RS:0;ubuntu:44139] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-01-13 19:32:18,124 INFO [ProcExecWrkr-2] assignment.AssignProcedure(211): Start pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2018-01-13 19:32:18,125 DEBUG [RS:0;ubuntu:44139] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2018-01-13 19:32:18,125 DEBUG [RS:0;ubuntu:44139] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2018-01-13 19:32:18,126 INFO [RS:1;ubuntu:40283] regionserver.HRegionServer(1535): Serving as ubuntu,40283,1515843136612, RpcServer on ubuntu/127.0.1.1:40283, sessionid=0x160ef4aa4c30002 2018-01-13 19:32:18,126 DEBUG [RS:1;ubuntu:40283] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2018-01-13 19:32:18,126 DEBUG [RS:2;ubuntu:38841] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2018-01-13 19:32:18,127 DEBUG [RS:2;ubuntu:38841] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager ubuntu,38841,1515843136669 2018-01-13 19:32:18,126 DEBUG [RS:0;ubuntu:44139] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager ubuntu,44139,1515843136509 2018-01-13 19:32:18,128 DEBUG [RS:0;ubuntu:44139] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'ubuntu,44139,1515843136509' 2018-01-13 19:32:18,128 DEBUG [RS:0;ubuntu:44139] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-01-13 19:32:18,128 DEBUG [RS:2;ubuntu:38841] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'ubuntu,38841,1515843136669' 2018-01-13 19:32:18,130 DEBUG [RS:2;ubuntu:38841] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-01-13 19:32:18,126 DEBUG [RS:1;ubuntu:40283] flush.RegionServerFlushTableProcedureManager(104): Start region server flush procedure manager ubuntu,40283,1515843136612 2018-01-13 19:32:18,130 DEBUG [RS:1;ubuntu:40283] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'ubuntu,40283,1515843136612' 2018-01-13 19:32:18,130 DEBUG [RS:1;ubuntu:40283] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2018-01-13 19:32:18,130 DEBUG [RS:2;ubuntu:38841] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-01-13 19:32:18,130 DEBUG [RS:1;ubuntu:40283] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2018-01-13 19:32:18,131 DEBUG [RS:1;ubuntu:40283] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2018-01-13 19:32:18,131 DEBUG [RS:1;ubuntu:40283] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2018-01-13 19:32:18,130 DEBUG [RS:0;ubuntu:44139] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-01-13 19:32:18,131 DEBUG [RS:2;ubuntu:38841] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2018-01-13 19:32:18,134 DEBUG [RS:2;ubuntu:38841] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2018-01-13 19:32:18,134 DEBUG [RS:2;ubuntu:38841] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager ubuntu,38841,1515843136669 2018-01-13 19:32:18,134 DEBUG [RS:2;ubuntu:38841] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'ubuntu,38841,1515843136669' 2018-01-13 19:32:18,135 DEBUG [RS:2;ubuntu:38841] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-01-13 19:32:18,135 DEBUG [RS:0;ubuntu:44139] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2018-01-13 19:32:18,135 INFO [RS:0;ubuntu:44139] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-01-13 19:32:18,135 INFO [RS:0;ubuntu:44139] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-01-13 19:32:18,136 DEBUG [RS:2;ubuntu:38841] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-01-13 19:32:18,136 DEBUG [RS:2;ubuntu:38841] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2018-01-13 19:32:18,131 DEBUG [RS:1;ubuntu:40283] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager ubuntu,40283,1515843136612 2018-01-13 19:32:18,138 DEBUG [RS:1;ubuntu:40283] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'ubuntu,40283,1515843136612' 2018-01-13 19:32:18,138 DEBUG [RS:1;ubuntu:40283] procedure.ZKProcedureMemberRpcs(135): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2018-01-13 19:32:18,136 INFO [RS:2;ubuntu:38841] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-01-13 19:32:18,138 INFO [RS:2;ubuntu:38841] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-01-13 19:32:18,139 DEBUG [RS:1;ubuntu:40283] procedure.ZKProcedureMemberRpcs(151): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2018-01-13 19:32:18,139 DEBUG [RS:1;ubuntu:40283] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2018-01-13 19:32:18,139 INFO [RS:1;ubuntu:40283] quotas.RegionServerRpcQuotaManager(62): Quota support disabled 2018-01-13 19:32:18,140 INFO [RS:1;ubuntu:40283] quotas.RegionServerSpaceQuotaManager(74): Quota support disabled, not starting space quota manager. 2018-01-13 19:32:18,295 DEBUG [AssignmentThread] assignment.AssignmentManager(1758): Processing assignment plans for System tables sysServersCount=3, allServersCount=3 2018-01-13 19:32:18,306 DEBUG [AssignmentThread] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region hbase:meta,,1.1588230740 2018-01-13 19:32:18,335 INFO [ProcExecWrkr-3] assignment.AssignProcedure(239): Early suspend! pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OFFLINE, location=ubuntu,38841,1515843136669 2018-01-13 19:32:19,193 INFO [RS:1;ubuntu:40283] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=ubuntu%2C40283%2C1515843136612, suffix=, logDir=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/WALs/ubuntu,40283,1515843136612, archiveDir=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/oldWALs 2018-01-13 19:32:19,194 INFO [RS:0;ubuntu:44139] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=ubuntu%2C44139%2C1515843136509, suffix=, logDir=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/WALs/ubuntu,44139,1515843136509, archiveDir=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/oldWALs 2018-01-13 19:32:19,194 INFO [RS:2;ubuntu:38841] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=ubuntu%2C38841%2C1515843136669, suffix=, logDir=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/WALs/ubuntu,38841,1515843136669, archiveDir=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/oldWALs 2018-01-13 19:32:19,228 DEBUG [RS:0;ubuntu:44139] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(417): org.apache.hadoop.hdfs.protocolPB.PBHelperClient not found (Hadoop is pre-2.8.0?); using class org.apache.hadoop.hdfs.protocolPB.PBHelper instead. 2018-01-13 19:32:19,273 DEBUG [RS:0;ubuntu:44139] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(490): No DfsClientConf class found, should be hadoop 2.7- java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.client.impl.DfsClientConf at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createChecksumCreater(FanOutOneBlockAsyncDFSOutputHelper.java:488) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:552) at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:51) at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:159) at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:167) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:99) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:628) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:1) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:766) at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:504) at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.(AsyncFSWAL.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:66) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:1) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:1) at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:263) at org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:2119) at org.apache.hadoop.hbase.regionserver.HRegionServer.buildServerLoad(HRegionServer.java:1314) at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1195) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1007) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:172) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$0(MiniHBaseCluster.java:171) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:156) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1726) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:154) at java.lang.Thread.run(Thread.java:748) 2018-01-13 19:32:19,278 DEBUG [RS:0;ubuntu:44139] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(524): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2018-01-13 19:32:19,306 DEBUG [RS-EventLoopGroup-5-5] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(199): No PBHelperClient class found, should be hadoop 2.7- java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.protocolPB.PBHelperClient at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createPBHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:197) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:261) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:657) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$3(FanOutOneBlockAsyncDFSOutputHelper.java:652) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$14.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:716) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$14.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:1) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:638) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:676) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:552) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) 2018-01-13 19:32:19,316 DEBUG [RS-EventLoopGroup-5-5] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39981,DS-2682b908-d71d-4edd-a7d6-c00e22c135f0,DISK] 2018-01-13 19:32:19,316 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45841,DS-6e234590-c709-4272-a78c-d0a2418a7171,DISK] 2018-01-13 19:32:19,318 DEBUG [RS-EventLoopGroup-5-7] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39981,DS-2682b908-d71d-4edd-a7d6-c00e22c135f0,DISK] 2018-01-13 19:32:19,326 DEBUG [RS-EventLoopGroup-5-6] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35477,DS-7b74f716-787f-414f-bbbf-7d604b248220,DISK] 2018-01-13 19:32:19,328 DEBUG [RS-EventLoopGroup-5-4] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45841,DS-6e234590-c709-4272-a78c-d0a2418a7171,DISK] 2018-01-13 19:32:19,328 DEBUG [RS-EventLoopGroup-5-8] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35477,DS-3a12d4a8-59eb-4588-98a1-23de755fb453,DISK] 2018-01-13 19:32:19,333 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35477,DS-7b74f716-787f-414f-bbbf-7d604b248220,DISK] 2018-01-13 19:32:19,340 DEBUG [RS-EventLoopGroup-5-8] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39981,DS-2682b908-d71d-4edd-a7d6-c00e22c135f0,DISK] 2018-01-13 19:32:19,360 DEBUG [RS-EventLoopGroup-5-5] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45841,DS-2ae3c181-7033-4bb1-926b-79afb8ba9668,DISK] 2018-01-13 19:32:19,398 INFO [RS:2;ubuntu:38841] wal.AbstractFSWAL(708): New WAL /user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/WALs/ubuntu,38841,1515843136669/ubuntu%2C38841%2C1515843136669.1515843139217 2018-01-13 19:32:19,398 INFO [RS:1;ubuntu:40283] wal.AbstractFSWAL(708): New WAL /user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/WALs/ubuntu,40283,1515843136612/ubuntu%2C40283%2C1515843136612.1515843139217 2018-01-13 19:32:19,399 INFO [RS:0;ubuntu:44139] wal.AbstractFSWAL(708): New WAL /user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/WALs/ubuntu,44139,1515843136509/ubuntu%2C44139%2C1515843136509.1515843139217 2018-01-13 19:32:19,399 DEBUG [RS:2;ubuntu:38841] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35477,DS-7b74f716-787f-414f-bbbf-7d604b248220,DISK], DatanodeInfoWithStorage[127.0.0.1:45841,DS-6e234590-c709-4272-a78c-d0a2418a7171,DISK], DatanodeInfoWithStorage[127.0.0.1:39981,DS-2682b908-d71d-4edd-a7d6-c00e22c135f0,DISK]] 2018-01-13 19:32:19,399 DEBUG [RS:0;ubuntu:44139] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39981,DS-2682b908-d71d-4edd-a7d6-c00e22c135f0,DISK], DatanodeInfoWithStorage[127.0.0.1:35477,DS-3a12d4a8-59eb-4588-98a1-23de755fb453,DISK], DatanodeInfoWithStorage[127.0.0.1:45841,DS-2ae3c181-7033-4bb1-926b-79afb8ba9668,DISK]] 2018-01-13 19:32:19,399 DEBUG [RS:1;ubuntu:40283] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35477,DS-7b74f716-787f-414f-bbbf-7d604b248220,DISK], DatanodeInfoWithStorage[127.0.0.1:45841,DS-6e234590-c709-4272-a78c-d0a2418a7171,DISK], DatanodeInfoWithStorage[127.0.0.1:39981,DS-2682b908-d71d-4edd-a7d6-c00e22c135f0,DISK]] 2018-01-13 19:32:19,424 INFO [ProcExecWrkr-4] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as ubuntu,38841,1515843136669 2018-01-13 19:32:19,435 DEBUG [ProcExecWrkr-4] zookeeper.MetaTableLocator(466): META region location doesn't exist, create it 2018-01-13 19:32:19,437 INFO [ProcExecWrkr-4] assignment.RegionTransitionProcedure(208): Dispatch pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=ubuntu,38841,1515843136669 2018-01-13 19:32:19,591 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=ubuntu,38841,1515843136669 version=2097152 2018-01-13 19:32:19,646 DEBUG [RSProcedureDispatcher-pool4-t1] master.ServerManager(783): New admin connection to ubuntu,38841,1515843136669 2018-01-13 19:32:19,653 DEBUG [RSProcedureDispatcher-pool4-t1] ipc.NettyRpcConnection(255): Connecting to ubuntu/127.0.1.1:38841 2018-01-13 19:32:19,663 INFO [RS-EventLoopGroup-5-5] ipc.ServerRpcConnection(527): Auth successful for zhangduo (auth:SIMPLE) 2018-01-13 19:32:19,663 INFO [RS-EventLoopGroup-5-5] ipc.ServerRpcConnection(557): Connection from 127.0.0.1 port: 59480 with version info: version: "2.0.0-beta-1" url: "git://ubuntu/home/zhangduo/hbase/code" revision: "a90cf1bc134a14ba7b692c8eb6ee7cddc4b857bf" user: "zhangduo" date: "Fri Jan 12 21:37:44 CST 2018" src_checksum: "5c5be6fe364dffb36035e82ded054eaa" version_major: 2 version_minor: 0 2018-01-13 19:32:19,671 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=38841] regionserver.RSRpcServices(1907): Open hbase:meta,,1.1588230740 2018-01-13 19:32:19,676 INFO [RS_OPEN_META-ubuntu:38841-0] wal.WALFactory(140): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2018-01-13 19:32:19,687 INFO [RS_OPEN_META-ubuntu:38841-0] wal.AbstractFSWAL(425): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=ubuntu%2C38841%2C1515843136669.meta, suffix=.meta, logDir=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/WALs/ubuntu,38841,1515843136669, archiveDir=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/oldWALs 2018-01-13 19:32:19,699 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:39981,DS-c786ffef-0cfe-40db-9609-e37ce51fcb98,DISK] 2018-01-13 19:32:19,703 DEBUG [RS-EventLoopGroup-5-4] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35477,DS-7b74f716-787f-414f-bbbf-7d604b248220,DISK] 2018-01-13 19:32:19,710 DEBUG [RS-EventLoopGroup-5-8] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(737): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:45841,DS-2ae3c181-7033-4bb1-926b-79afb8ba9668,DISK] 2018-01-13 19:32:19,722 INFO [RS_OPEN_META-ubuntu:38841-0] wal.AbstractFSWAL(708): New WAL /user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/WALs/ubuntu,38841,1515843136669/ubuntu%2C38841%2C1515843136669.meta.1515843139688.meta 2018-01-13 19:32:19,723 DEBUG [RS_OPEN_META-ubuntu:38841-0] wal.AbstractFSWAL(772): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:39981,DS-c786ffef-0cfe-40db-9609-e37ce51fcb98,DISK], DatanodeInfoWithStorage[127.0.0.1:35477,DS-7b74f716-787f-414f-bbbf-7d604b248220,DISK], DatanodeInfoWithStorage[127.0.0.1:45841,DS-2ae3c181-7033-4bb1-926b-79afb8ba9668,DISK]] 2018-01-13 19:32:19,723 DEBUG [RS_OPEN_META-ubuntu:38841-0] regionserver.HRegion(6949): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2018-01-13 19:32:19,745 DEBUG [RS_OPEN_META-ubuntu:38841-0] coprocessor.CoprocessorHost(201): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2018-01-13 19:32:19,759 DEBUG [RS_OPEN_META-ubuntu:38841-0] regionserver.HRegion(7902): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2018-01-13 19:32:19,768 INFO [RS_OPEN_META-ubuntu:38841-0] regionserver.RegionCoprocessorHost(427): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2018-01-13 19:32:19,772 DEBUG [RS_OPEN_META-ubuntu:38841-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table meta 1588230740 2018-01-13 19:32:19,773 DEBUG [RS_OPEN_META-ubuntu:38841-0] regionserver.HRegion(807): Instantiated hbase:meta,,1.1588230740 2018-01-13 19:32:19,782 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/info 2018-01-13 19:32:19,782 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/info 2018-01-13 19:32:19,782 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 19:32:19,783 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=820320, freeSize=1117800608, maxSize=1118620928, heapSize=820320, minSize=1062689856, minFactor=0.95, multiSize=531344928, multiFactor=0.5, singleSize=265672464, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 19:32:19,783 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 19:32:19,786 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/rep_barrier 2018-01-13 19:32:19,787 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/rep_barrier 2018-01-13 19:32:19,787 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 19:32:19,787 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=820320, freeSize=1117800608, maxSize=1118620928, heapSize=820320, minSize=1062689856, minFactor=0.95, multiSize=531344928, multiFactor=0.5, singleSize=265672464, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 19:32:19,787 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 19:32:19,791 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/rep_meta 2018-01-13 19:32:19,791 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/rep_meta 2018-01-13 19:32:19,791 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 19:32:19,791 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_meta: blockCache=LruBlockCache{blockCount=0, currentSize=820320, freeSize=1117800608, maxSize=1118620928, heapSize=820320, minSize=1062689856, minFactor=0.95, multiSize=531344928, multiFactor=0.5, singleSize=265672464, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 19:32:19,792 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 19:32:19,795 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/rep_position 2018-01-13 19:32:19,795 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/rep_position 2018-01-13 19:32:19,796 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 19:32:19,796 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for rep_position: blockCache=LruBlockCache{blockCount=0, currentSize=820320, freeSize=1117800608, maxSize=1118620928, heapSize=820320, minSize=1062689856, minFactor=0.95, multiSize=531344928, multiFactor=0.5, singleSize=265672464, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 19:32:19,796 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 19:32:19,799 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/table 2018-01-13 19:32:19,800 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/table 2018-01-13 19:32:19,800 INFO [StoreOpener-1588230740-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 19:32:19,800 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=820320, freeSize=1117800608, maxSize=1118620928, heapSize=820320, minSize=1062689856, minFactor=0.95, multiSize=531344928, multiFactor=0.5, singleSize=265672464, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 19:32:19,801 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 19:32:19,807 DEBUG [RS_OPEN_META-ubuntu:38841-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740 2018-01-13 19:32:19,819 DEBUG [RS_OPEN_META-ubuntu:38841-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushSize/# of families (25.6M)) instead. 2018-01-13 19:32:19,826 DEBUG [RS_OPEN_META-ubuntu:38841-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/recovered.edits/3.seqid, newSeqId=3, maxSeqId=2 2018-01-13 19:32:19,827 INFO [RS_OPEN_META-ubuntu:38841-0] regionserver.HRegion(960): Onlined 1588230740; next sequenceid=3 2018-01-13 19:32:19,867 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2197): Post open deploy tasks for hbase:meta,,1.1588230740 2018-01-13 19:32:19,895 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=36079] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=3, pid=2, ppid=1, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1588230740; rit=OPENING, location=ubuntu,38841,1515843136669 2018-01-13 19:32:19,899 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(2221): Finished post open deploy task for hbase:meta,,1.1588230740 2018-01-13 19:32:19,902 DEBUG [RS_OPEN_META-ubuntu:38841-0] handler.OpenRegionHandler(128): Opened hbase:meta,,1.1588230740 on ubuntu,38841,1515843136669 2018-01-13 19:32:19,949 DEBUG [ProcExecWrkr-5] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:meta","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515843139903}]},"ts":1515843139903} 2018-01-13 19:32:19,961 WARN [ProcExecWrkr-5] client.ZKAsyncRegistry(168): Failed to fetch /hbase/meta-region-server-1 org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/meta-region-server-1 at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:164) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:321) at java.lang.Thread.run(Thread.java:748) 2018-01-13 19:32:19,962 WARN [ReadOnlyZKClient] client.ZKAsyncRegistry(168): Failed to fetch /hbase/meta-region-server-2 org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/meta-region-server-2 at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:164) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:321) at java.lang.Thread.run(Thread.java:748) 2018-01-13 19:32:19,990 DEBUG [hconnection-0x4e18358f-shared-pool8-t1] ipc.NettyRpcConnection(255): Connecting to ubuntu/127.0.1.1:38841 2018-01-13 19:32:20,004 INFO [RS-EventLoopGroup-5-5] ipc.ServerRpcConnection(527): Auth successful for zhangduo (auth:SIMPLE) 2018-01-13 19:32:20,004 INFO [RS-EventLoopGroup-5-5] ipc.ServerRpcConnection(557): Connection from 127.0.0.1 port: 59488 with version info: version: "2.0.0-beta-1" url: "git://ubuntu/home/zhangduo/hbase/code" revision: "a90cf1bc134a14ba7b692c8eb6ee7cddc4b857bf" user: "zhangduo" date: "Fri Jan 12 21:37:44 CST 2018" src_checksum: "5c5be6fe364dffb36035e82ded054eaa" version_major: 2 version_minor: 0 2018-01-13 19:32:20,051 INFO [ProcExecWrkr-5] hbase.MetaTableAccessor(1786): Updated table hbase:meta state to ENABLED in META 2018-01-13 19:32:20,052 INFO [ProcExecWrkr-5] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=0) location in ZooKeeper as ubuntu,38841,1515843136669 2018-01-13 19:32:20,055 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2018-01-13 19:32:20,060 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=2, ppid=1, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 2018-01-13 19:32:20,064 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=1, state=RUNNABLE; RecoverMetaProcedure failedMetaServer=null, splitWal=true; resume parent processing. 2018-01-13 19:32:20,064 INFO [ProcExecWrkr-5] procedure2.ProcedureExecutor(1250): Finished pid=2, ppid=1, state=SUCCESS; AssignProcedure table=hbase:meta, region=1588230740 in 1.9590sec 2018-01-13 19:32:20,176 INFO [ProcExecWrkr-6] procedure2.ProcedureExecutor(1250): Finished pid=1, state=SUCCESS; RecoverMetaProcedure failedMetaServer=null, splitWal=true in 2.2480sec 2018-01-13 19:32:20,178 INFO [M:0;ubuntu:36079] master.HMaster(3495): hbase:meta (default replica) deployed at=ubuntu,38841,1515843136669 2018-01-13 19:32:20,266 INFO [M:0;ubuntu:36079] master.MasterWalManager(189): Log folder hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/WALs/ubuntu,38841,1515843136669 belongs to an existing region server 2018-01-13 19:32:20,267 INFO [M:0;ubuntu:36079] master.MasterWalManager(189): Log folder hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/WALs/ubuntu,40283,1515843136612 belongs to an existing region server 2018-01-13 19:32:20,268 INFO [M:0;ubuntu:36079] master.MasterWalManager(189): Log folder hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/WALs/ubuntu,44139,1515843136509 belongs to an existing region server 2018-01-13 19:32:20,268 INFO [M:0;ubuntu:36079] assignment.AssignmentManager(1202): Joining cluster...Loading hbase:meta content. 2018-01-13 19:32:20,274 INFO [M:0;ubuntu:36079] assignment.AssignmentManager(1212): Number of RegionServers=3 2018-01-13 19:32:20,275 INFO [M:0;ubuntu:36079] procedure2.ProcedureExecutor$TimeoutExecutorThread(1827): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=5000, timestamp=1515843145275 2018-01-13 19:32:20,275 INFO [M:0;ubuntu:36079] assignment.AssignmentManager(1219): Joined the cluster in 7msec, failover=true 2018-01-13 19:32:20,316 INFO [M:0;ubuntu:36079] master.TableNamespaceManager(93): Namespace table not found. Creating... 2018-01-13 19:32:20,321 INFO [M:0;ubuntu:36079] master.HMaster(1776): Client=null/null create 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} 2018-01-13 19:32:20,460 DEBUG [M:0;ubuntu:36079] procedure2.ProcedureExecutor(869): Stored pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 19:32:20,465 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 19:32:20,487 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT; CreateTableProcedure table=hbase:namespace 2018-01-13 19:32:20,541 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45841 is added to blk_1073741834_1010{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-3a12d4a8-59eb-4588-98a1-23de755fb453:NORMAL:127.0.0.1:35477|RBW], ReplicaUC[[DISK]DS-c786ffef-0cfe-40db-9609-e37ce51fcb98:NORMAL:127.0.0.1:39981|RBW], ReplicaUC[[DISK]DS-2ae3c181-7033-4bb1-926b-79afb8ba9668:NORMAL:127.0.0.1:45841|RBW]]} size 0 2018-01-13 19:32:20,543 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39981 is added to blk_1073741834_1010{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-3a12d4a8-59eb-4588-98a1-23de755fb453:NORMAL:127.0.0.1:35477|RBW], ReplicaUC[[DISK]DS-2ae3c181-7033-4bb1-926b-79afb8ba9668:NORMAL:127.0.0.1:45841|RBW], ReplicaUC[[DISK]DS-2682b908-d71d-4edd-a7d6-c00e22c135f0:NORMAL:127.0.0.1:39981|FINALIZED]]} size 0 2018-01-13 19:32:20,548 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35477 is added to blk_1073741834_1010 size 476 2018-01-13 19:32:20,551 DEBUG [ProcExecWrkr-7] util.FSTableDescriptors(716): Wrote descriptor into: hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2018-01-13 19:32:20,554 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(6772): creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} RootDir = hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/.tmp Table name == hbase:namespace 2018-01-13 19:32:20,579 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35477 is added to blk_1073741835_1011{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-6e234590-c709-4272-a78c-d0a2418a7171:NORMAL:127.0.0.1:45841|RBW], ReplicaUC[[DISK]DS-2682b908-d71d-4edd-a7d6-c00e22c135f0:NORMAL:127.0.0.1:39981|RBW], ReplicaUC[[DISK]DS-7b74f716-787f-414f-bbbf-7d604b248220:NORMAL:127.0.0.1:35477|FINALIZED]]} size 0 2018-01-13 19:32:20,580 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39981 is added to blk_1073741835_1011{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-6e234590-c709-4272-a78c-d0a2418a7171:NORMAL:127.0.0.1:45841|RBW], ReplicaUC[[DISK]DS-7b74f716-787f-414f-bbbf-7d604b248220:NORMAL:127.0.0.1:35477|FINALIZED], ReplicaUC[[DISK]DS-c786ffef-0cfe-40db-9609-e37ce51fcb98:NORMAL:127.0.0.1:39981|FINALIZED]]} size 0 2018-01-13 19:32:20,582 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45841 is added to blk_1073741835_1011{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-6e234590-c709-4272-a78c-d0a2418a7171:NORMAL:127.0.0.1:45841|RBW], ReplicaUC[[DISK]DS-7b74f716-787f-414f-bbbf-7d604b248220:NORMAL:127.0.0.1:35477|FINALIZED], ReplicaUC[[DISK]DS-c786ffef-0cfe-40db-9609-e37ce51fcb98:NORMAL:127.0.0.1:39981|FINALIZED]]} size 0 2018-01-13 19:32:20,583 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(807): Instantiated hbase:namespace,,1515843140318.29330989a4082382160ba4b6b47c5e66. 2018-01-13 19:32:20,584 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1481): Closing hbase:namespace,,1515843140318.29330989a4082382160ba4b6b47c5e66.: disabling compactions & flushes 2018-01-13 19:32:20,584 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1520): Updates disabled for region hbase:namespace,,1515843140318.29330989a4082382160ba4b6b47c5e66. 2018-01-13 19:32:20,585 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1633): Closed hbase:namespace,,1515843140318.29330989a4082382160ba4b6b47c5e66. 2018-01-13 19:32:20,600 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_ADD_TO_META; CreateTableProcedure table=hbase:namespace 2018-01-13 19:32:20,611 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(2131): Put {"totalColumns":2,"row":"hbase:namespace,,1515843140318.29330989a4082382160ba4b6b47c5e66.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":9223372036854775807},{"qualifier":"state","vlen":6,"tag":[],"timestamp":9223372036854775807}]},"ts":9223372036854775807} 2018-01-13 19:32:20,618 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1638): Added 1 regions to meta. 2018-01-13 19:32:20,624 DEBUG [ProcExecWrkr-7] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=hbase:namespace 2018-01-13 19:32:20,624 DEBUG [ProcExecWrkr-7] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515843140624}]},"ts":1515843140624} 2018-01-13 19:32:20,633 INFO [ProcExecWrkr-7] hbase.MetaTableAccessor(1786): Updated table hbase:namespace state to ENABLING in META 2018-01-13 19:32:20,634 DEBUG [ProcExecWrkr-7] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region hbase:namespace,,1515843140318.29330989a4082382160ba4b6b47c5e66. 2018-01-13 19:32:20,645 INFO [ProcExecWrkr-7] procedure2.ProcedureExecutor(1499): Initialized subprocedures=[{pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=29330989a4082382160ba4b6b47c5e66, target=ubuntu,44139,1515843136509}] 2018-01-13 19:32:20,654 INFO [ProcExecWrkr-9] procedure.MasterProcedureScheduler(883): pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=29330989a4082382160ba4b6b47c5e66, target=ubuntu,44139,1515843136509 hbase:namespace hbase:namespace,,1515843140318.29330989a4082382160ba4b6b47c5e66. 2018-01-13 19:32:20,663 INFO [ProcExecWrkr-9] assignment.AssignProcedure(211): Start pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:namespace, region=29330989a4082382160ba4b6b47c5e66, target=ubuntu,44139,1515843136509; rit=OFFLINE, location=ubuntu,44139,1515843136509; forceNewPlan=false, retain=false 2018-01-13 19:32:20,839 INFO [AssignmentThread] balancer.BaseLoadBalancer(1470): Reassigned 1 regions. 1 retained the pre-restart assignment. 2018-01-13 19:32:20,843 INFO [ProcExecWrkr-8] assignment.RegionStateStore(201): pid=4 updating hbase:meta row=hbase:namespace,,1515843140318.29330989a4082382160ba4b6b47c5e66., regionState=OPENING, regionLocation=ubuntu,44139,1515843136509 2018-01-13 19:32:20,847 INFO [ProcExecWrkr-8] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x479e1131 to localhost:53209, session timeout 90000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 19:32:20,861 DEBUG [ProcExecWrkr-8] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a31b0a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 19:32:20,864 WARN [ReadOnlyZKClient] client.ZKAsyncRegistry(168): Failed to fetch /hbase/meta-region-server-1 org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/meta-region-server-1 at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:164) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:321) at java.lang.Thread.run(Thread.java:748) 2018-01-13 19:32:20,865 WARN [ReadOnlyZKClient] client.ZKAsyncRegistry(168): Failed to fetch /hbase/meta-region-server-2 org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/meta-region-server-2 at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:164) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:321) at java.lang.Thread.run(Thread.java:748) 2018-01-13 19:32:20,868 DEBUG [MultiHConnection-shared--pool9-t1] ipc.NettyRpcConnection(255): Connecting to ubuntu/127.0.1.1:38841 2018-01-13 19:32:20,896 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(527): Auth successful for zhangduo (auth:SIMPLE) 2018-01-13 19:32:20,897 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(557): Connection from 127.0.0.1 port: 59506 with version info: version: "2.0.0-beta-1" url: "git://ubuntu/home/zhangduo/hbase/code" revision: "a90cf1bc134a14ba7b692c8eb6ee7cddc4b857bf" user: "zhangduo" date: "Fri Jan 12 21:37:44 CST 2018" src_checksum: "5c5be6fe364dffb36035e82ded054eaa" version_major: 2 version_minor: 0 2018-01-13 19:32:20,920 INFO [ProcExecWrkr-8] assignment.RegionTransitionProcedure(208): Dispatch pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=29330989a4082382160ba4b6b47c5e66, target=ubuntu,44139,1515843136509; rit=OPENING, location=ubuntu,44139,1515843136509 2018-01-13 19:32:21,072 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=ubuntu,44139,1515843136509 version=2097152 2018-01-13 19:32:21,073 DEBUG [RSProcedureDispatcher-pool4-t2] master.ServerManager(783): New admin connection to ubuntu,44139,1515843136509 2018-01-13 19:32:21,074 DEBUG [RSProcedureDispatcher-pool4-t2] ipc.NettyRpcConnection(255): Connecting to ubuntu/127.0.1.1:44139 2018-01-13 19:32:21,084 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(527): Auth successful for zhangduo (auth:SIMPLE) 2018-01-13 19:32:21,085 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(557): Connection from 127.0.0.1 port: 52046 with version info: version: "2.0.0-beta-1" url: "git://ubuntu/home/zhangduo/hbase/code" revision: "a90cf1bc134a14ba7b692c8eb6ee7cddc4b857bf" user: "zhangduo" date: "Fri Jan 12 21:37:44 CST 2018" src_checksum: "5c5be6fe364dffb36035e82ded054eaa" version_major: 2 version_minor: 0 2018-01-13 19:32:21,087 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=44139] regionserver.RSRpcServices(1907): Open hbase:namespace,,1515843140318.29330989a4082382160ba4b6b47c5e66. 2018-01-13 19:32:21,101 DEBUG [RS_OPEN_PRIORITY_REGION-ubuntu:44139-0] regionserver.HRegion(6949): Opening region: {ENCODED => 29330989a4082382160ba4b6b47c5e66, NAME => 'hbase:namespace,,1515843140318.29330989a4082382160ba4b6b47c5e66.', STARTKEY => '', ENDKEY => ''} 2018-01-13 19:32:21,103 DEBUG [RS_OPEN_PRIORITY_REGION-ubuntu:44139-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table namespace 29330989a4082382160ba4b6b47c5e66 2018-01-13 19:32:21,103 DEBUG [RS_OPEN_PRIORITY_REGION-ubuntu:44139-0] regionserver.HRegion(807): Instantiated hbase:namespace,,1515843140318.29330989a4082382160ba4b6b47c5e66. 2018-01-13 19:32:21,112 DEBUG [StoreOpener-29330989a4082382160ba4b6b47c5e66-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/namespace/29330989a4082382160ba4b6b47c5e66/info 2018-01-13 19:32:21,112 DEBUG [StoreOpener-29330989a4082382160ba4b6b47c5e66-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/namespace/29330989a4082382160ba4b6b47c5e66/info 2018-01-13 19:32:21,112 INFO [StoreOpener-29330989a4082382160ba4b6b47c5e66-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 19:32:21,112 INFO [StoreOpener-29330989a4082382160ba4b6b47c5e66-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=820320, freeSize=1117800608, maxSize=1118620928, heapSize=820320, minSize=1062689856, minFactor=0.95, multiSize=531344928, multiFactor=0.5, singleSize=265672464, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 19:32:21,114 INFO [StoreOpener-29330989a4082382160ba4b6b47c5e66-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 19:32:21,120 DEBUG [RS_OPEN_PRIORITY_REGION-ubuntu:44139-0] regionserver.HRegion(4350): Found 0 recovered edits file(s) under hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/namespace/29330989a4082382160ba4b6b47c5e66 2018-01-13 19:32:21,134 DEBUG [RS_OPEN_PRIORITY_REGION-ubuntu:44139-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/namespace/29330989a4082382160ba4b6b47c5e66/recovered.edits/2.seqid, newSeqId=2, maxSeqId=0 2018-01-13 19:32:21,134 INFO [RS_OPEN_PRIORITY_REGION-ubuntu:44139-0] regionserver.HRegion(960): Onlined 29330989a4082382160ba4b6b47c5e66; next sequenceid=2 2018-01-13 19:32:21,138 INFO [PostOpenDeployTasks:29330989a4082382160ba4b6b47c5e66] regionserver.HRegionServer(2197): Post open deploy tasks for hbase:namespace,,1515843140318.29330989a4082382160ba4b6b47c5e66. 2018-01-13 19:32:21,157 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=36079] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=4, ppid=3, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:namespace, region=29330989a4082382160ba4b6b47c5e66, target=ubuntu,44139,1515843136509; rit=OPENING, location=ubuntu,44139,1515843136509 2018-01-13 19:32:21,159 INFO [ProcExecWrkr-10] assignment.RegionStateStore(201): pid=4 updating hbase:meta row=hbase:namespace,,1515843140318.29330989a4082382160ba4b6b47c5e66., regionState=OPEN, openSeqNum=2, regionLocation=ubuntu,44139,1515843136509 2018-01-13 19:32:21,160 DEBUG [PostOpenDeployTasks:29330989a4082382160ba4b6b47c5e66] regionserver.HRegionServer(2221): Finished post open deploy task for hbase:namespace,,1515843140318.29330989a4082382160ba4b6b47c5e66. 2018-01-13 19:32:21,161 DEBUG [RS_OPEN_PRIORITY_REGION-ubuntu:44139-0] handler.OpenRegionHandler(128): Opened hbase:namespace,,1515843140318.29330989a4082382160ba4b6b47c5e66. on ubuntu,44139,1515843136509 2018-01-13 19:32:21,175 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1613): Finish suprocedure pid=4, ppid=3, state=SUCCESS; AssignProcedure table=hbase:namespace, region=29330989a4082382160ba4b6b47c5e66, target=ubuntu,44139,1515843136509 2018-01-13 19:32:21,181 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1619): Finished subprocedure(s) of pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace; resume parent processing. 2018-01-13 19:32:21,182 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE; CreateTableProcedure table=hbase:namespace 2018-01-13 19:32:21,182 INFO [ProcExecWrkr-10] procedure2.ProcedureExecutor(1250): Finished pid=4, ppid=3, state=SUCCESS; AssignProcedure table=hbase:namespace, region=29330989a4082382160ba4b6b47c5e66, target=ubuntu,44139,1515843136509 in 527msec 2018-01-13 19:32:21,182 DEBUG [ProcExecWrkr-11] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515843141182}]},"ts":1515843141182} 2018-01-13 19:32:21,189 INFO [ProcExecWrkr-11] hbase.MetaTableAccessor(1786): Updated table hbase:namespace state to ENABLED in META 2018-01-13 19:32:21,194 DEBUG [ProcExecWrkr-11] procedure2.StateMachineProcedure(180): pid=3, state=RUNNABLE:CREATE_TABLE_POST_OPERATION; CreateTableProcedure table=hbase:namespace 2018-01-13 19:32:21,199 INFO [ProcExecWrkr-11] procedure2.ProcedureExecutor(1250): Finished pid=3, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 873msec 2018-01-13 19:32:21,222 DEBUG [M:0;ubuntu:36079] zookeeper.ZKUtil(357): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2018-01-13 19:32:21,224 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2018-01-13 19:32:21,239 DEBUG [M:0;ubuntu:36079] ipc.NettyRpcConnection(255): Connecting to ubuntu/127.0.1.1:44139 2018-01-13 19:32:21,252 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(527): Auth successful for zhangduo (auth:SIMPLE) 2018-01-13 19:32:21,252 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(557): Connection from 127.0.0.1 port: 52048 with version info: version: "2.0.0-beta-1" url: "git://ubuntu/home/zhangduo/hbase/code" revision: "a90cf1bc134a14ba7b692c8eb6ee7cddc4b857bf" user: "zhangduo" date: "Fri Jan 12 21:37:44 CST 2018" src_checksum: "5c5be6fe364dffb36035e82ded054eaa" version_major: 2 version_minor: 0 2018-01-13 19:32:21,383 DEBUG [M:0;ubuntu:36079] procedure2.ProcedureExecutor(869): Stored pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-01-13 19:32:21,386 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2018-01-13 19:32:21,409 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_CREATE_DIRECTORY; CreateNamespaceProcedure, namespace=default 2018-01-13 19:32:21,424 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_INSERT_INTO_NS_TABLE; CreateNamespaceProcedure, namespace=default 2018-01-13 19:32:21,484 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_UPDATE_ZK; CreateNamespaceProcedure, namespace=default 2018-01-13 19:32:21,486 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-13 19:32:21,494 DEBUG [ProcExecWrkr-12] procedure2.StateMachineProcedure(180): pid=5, state=RUNNABLE:CREATE_NAMESPACE_SET_NAMESPACE_QUOTA; CreateNamespaceProcedure, namespace=default 2018-01-13 19:32:21,508 INFO [ProcExecWrkr-12] procedure2.ProcedureExecutor(1250): Finished pid=5, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 234msec 2018-01-13 19:32:21,630 DEBUG [M:0;ubuntu:36079] procedure2.ProcedureExecutor(869): Stored pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 19:32:21,633 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 19:32:21,650 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_CREATE_DIRECTORY; CreateNamespaceProcedure, namespace=hbase 2018-01-13 19:32:21,658 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_INSERT_INTO_NS_TABLE; CreateNamespaceProcedure, namespace=hbase 2018-01-13 19:32:21,669 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_UPDATE_ZK; CreateNamespaceProcedure, namespace=hbase 2018-01-13 19:32:21,671 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2018-01-13 19:32:21,680 DEBUG [ProcExecWrkr-13] procedure2.StateMachineProcedure(180): pid=6, state=RUNNABLE:CREATE_NAMESPACE_SET_NAMESPACE_QUOTA; CreateNamespaceProcedure, namespace=hbase 2018-01-13 19:32:21,686 INFO [ProcExecWrkr-13] procedure2.ProcedureExecutor(1250): Finished pid=6, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 166msec 2018-01-13 19:32:21,705 DEBUG [M:0;ubuntu:36079] zookeeper.RecoverableZooKeeper(571): Node /hbase/namespace/default already exists 2018-01-13 19:32:21,712 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2018-01-13 19:32:21,714 DEBUG [M:0;ubuntu:36079] zookeeper.RecoverableZooKeeper(571): Node /hbase/namespace/hbase already exists 2018-01-13 19:32:21,716 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2018-01-13 19:32:21,717 INFO [M:0;ubuntu:36079] master.HMaster(928): Master has completed initialization 4.948sec 2018-01-13 19:32:21,721 DEBUG [M:0;ubuntu:36079] zookeeper.ZKUtil(614): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server-1 because node does not exist (not an error) 2018-01-13 19:32:21,722 DEBUG [M:0;ubuntu:36079] master.MasterMetaBootstrap(130): meta state from zookeeper: {534574363 state=OFFLINE, ts=1515843141722, server=null} 2018-01-13 19:32:21,722 DEBUG [M:0;ubuntu:36079] assignment.AssignmentManager(467): Assigning hbase:meta,,1_0001.534574363 2018-01-13 19:32:21,747 INFO [main] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x2aa7399c to localhost:53209, session timeout 90000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 19:32:21,758 DEBUG [main] ipc.AbstractRpcClient(200): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@11eed657, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2018-01-13 19:32:21,771 WARN [ReadOnlyZKClient] client.ZKAsyncRegistry(168): Failed to fetch /hbase/meta-region-server-1 org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/meta-region-server-1 at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:164) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:321) at java.lang.Thread.run(Thread.java:748) 2018-01-13 19:32:21,772 WARN [ReadOnlyZKClient] client.ZKAsyncRegistry(168): Failed to fetch /hbase/meta-region-server-2 org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/meta-region-server-2 at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:164) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:321) at java.lang.Thread.run(Thread.java:748) 2018-01-13 19:32:21,773 DEBUG [hconnection-0x69afa141-shared-pool11-t1] ipc.NettyRpcConnection(255): Connecting to ubuntu/127.0.1.1:38841 2018-01-13 19:32:21,787 INFO [RS-EventLoopGroup-5-7] ipc.ServerRpcConnection(527): Auth successful for zhangduo (auth:SIMPLE) 2018-01-13 19:32:21,787 INFO [RS-EventLoopGroup-5-7] ipc.ServerRpcConnection(557): Connection from 127.0.0.1 port: 59514 with version info: version: "2.0.0-beta-1" url: "git://ubuntu/home/zhangduo/hbase/code" revision: "a90cf1bc134a14ba7b692c8eb6ee7cddc4b857bf" user: "zhangduo" date: "Fri Jan 12 21:37:44 CST 2018" src_checksum: "5c5be6fe364dffb36035e82ded054eaa" version_major: 2 version_minor: 0 2018-01-13 19:32:21,791 INFO [main] hbase.HBaseTestingUtility(1036): Minicluster is up; activeMaster=ubuntu,36079,1515843135537 2018-01-13 19:32:21,792 INFO [main] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x52d6d273 to localhost:53209, session timeout 90000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 19:32:21,798 INFO [main] client.TestZKAsyncRegistry(102): STARTED TEST 2018-01-13 19:32:21,817 INFO [main] hbase.Waiter(189): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2018-01-13 19:32:21,819 WARN [ReadOnlyZKClient] client.ZKAsyncRegistry(168): Failed to fetch /hbase/meta-region-server-1 org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/meta-region-server-1 at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:164) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:321) at java.lang.Thread.run(Thread.java:748) 2018-01-13 19:32:21,819 WARN [ReadOnlyZKClient] client.ZKAsyncRegistry(168): Failed to fetch /hbase/meta-region-server-2 org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/meta-region-server-2 at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:164) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:321) at java.lang.Thread.run(Thread.java:748) 2018-01-13 19:32:21,828 DEBUG [M:0;ubuntu:36079] procedure2.ProcedureExecutor(869): Stored pid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=534574363 2018-01-13 19:32:21,831 INFO [ProcExecWrkr-14] procedure.MasterProcedureScheduler(883): pid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=534574363 hbase:meta hbase:meta,,1_0001.534574363 2018-01-13 19:32:21,831 INFO [ProcExecWrkr-14] assignment.AssignProcedure(211): Start pid=7, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=534574363; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2018-01-13 19:32:21,880 DEBUG [M:0;ubuntu:36079] procedure.ProcedureSyncWait(189): waitFor pid=7 2018-01-13 19:32:21,981 DEBUG [AssignmentThread] assignment.AssignmentManager(1758): Processing assignment plans for System tables sysServersCount=3, allServersCount=3 2018-01-13 19:32:21,982 DEBUG [AssignmentThread] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region hbase:meta,,1_0001.534574363 2018-01-13 19:32:21,993 INFO [ProcExecWrkr-15] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=1) location in ZooKeeper as ubuntu,44139,1515843136509 2018-01-13 19:32:21,995 DEBUG [ProcExecWrkr-15] zookeeper.MetaTableLocator(468): META region location doesn't exist for replicaId=1, create it 2018-01-13 19:32:21,997 INFO [ProcExecWrkr-15] assignment.RegionTransitionProcedure(208): Dispatch pid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=534574363; rit=OPENING, location=ubuntu,44139,1515843136509 2018-01-13 19:32:22,022 WARN [ReadOnlyZKClient] client.ZKAsyncRegistry(168): Failed to fetch /hbase/meta-region-server-2 org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/meta-region-server-2 at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:164) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:321) at java.lang.Thread.run(Thread.java:748) 2018-01-13 19:32:22,149 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=ubuntu,44139,1515843136509 version=2097152 2018-01-13 19:32:22,155 INFO [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=44139] regionserver.RSRpcServices(1907): Open hbase:meta,,1_0001.534574363 2018-01-13 19:32:22,160 DEBUG [RS_OPEN_META-ubuntu:44139-0] regionserver.HRegion(6949): Opening region: {ENCODED => 534574363, NAME => 'hbase:meta,,1_0001', STARTKEY => '', ENDKEY => '', REPLICA_ID => 1} 2018-01-13 19:32:22,161 DEBUG [RS_OPEN_META-ubuntu:44139-0] coprocessor.CoprocessorHost(201): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2018-01-13 19:32:22,162 DEBUG [RS_OPEN_META-ubuntu:44139-0] regionserver.HRegion(7902): Registered coprocessor service: region=hbase:meta,,1_0001 service=MultiRowMutationService 2018-01-13 19:32:22,165 INFO [RS_OPEN_META-ubuntu:44139-0] regionserver.RegionCoprocessorHost(427): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2018-01-13 19:32:22,166 DEBUG [RS_OPEN_META-ubuntu:44139-0] regionserver.MetricsRegionSourceImpl(75): Creating new MetricsRegionSourceImpl for table meta 534574363 2018-01-13 19:32:22,166 DEBUG [RS_OPEN_META-ubuntu:44139-0] regionserver.HRegion(807): Instantiated hbase:meta,,1_0001.534574363 2018-01-13 19:32:22,167 DEBUG [RS_OPEN_META-ubuntu:44139-0] regionserver.HRegion(895): Skipping creation of .regioninfo file for {ENCODED => 534574363, NAME => 'hbase:meta,,1_0001', STARTKEY => '', ENDKEY => '', REPLICA_ID => 1} 2018-01-13 19:32:22,175 DEBUG [StoreOpener-534574363-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/info 2018-01-13 19:32:22,175 DEBUG [StoreOpener-534574363-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/info 2018-01-13 19:32:22,176 INFO [StoreOpener-534574363-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 19:32:22,176 INFO [StoreOpener-534574363-1] hfile.CacheConfig(239): Created cacheConfig for info: blockCache=LruBlockCache{blockCount=0, currentSize=820320, freeSize=1117800608, maxSize=1118620928, heapSize=820320, minSize=1062689856, minFactor=0.95, multiSize=531344928, multiFactor=0.5, singleSize=265672464, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 19:32:22,177 INFO [StoreOpener-534574363-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 19:32:22,186 DEBUG [StoreOpener-534574363-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/rep_barrier 2018-01-13 19:32:22,187 DEBUG [StoreOpener-534574363-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/rep_barrier 2018-01-13 19:32:22,188 INFO [StoreOpener-534574363-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 19:32:22,188 INFO [StoreOpener-534574363-1] hfile.CacheConfig(239): Created cacheConfig for rep_barrier: blockCache=LruBlockCache{blockCount=0, currentSize=820320, freeSize=1117800608, maxSize=1118620928, heapSize=820320, minSize=1062689856, minFactor=0.95, multiSize=531344928, multiFactor=0.5, singleSize=265672464, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 19:32:22,189 INFO [StoreOpener-534574363-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 19:32:22,199 DEBUG [StoreOpener-534574363-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/rep_meta 2018-01-13 19:32:22,200 DEBUG [StoreOpener-534574363-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/rep_meta 2018-01-13 19:32:22,201 INFO [StoreOpener-534574363-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 19:32:22,201 INFO [StoreOpener-534574363-1] hfile.CacheConfig(239): Created cacheConfig for rep_meta: blockCache=LruBlockCache{blockCount=0, currentSize=820320, freeSize=1117800608, maxSize=1118620928, heapSize=820320, minSize=1062689856, minFactor=0.95, multiSize=531344928, multiFactor=0.5, singleSize=265672464, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 19:32:22,203 INFO [StoreOpener-534574363-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 19:32:22,207 DEBUG [StoreOpener-534574363-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/rep_position 2018-01-13 19:32:22,207 DEBUG [StoreOpener-534574363-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/rep_position 2018-01-13 19:32:22,208 INFO [StoreOpener-534574363-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 19:32:22,208 INFO [StoreOpener-534574363-1] hfile.CacheConfig(239): Created cacheConfig for rep_position: blockCache=LruBlockCache{blockCount=0, currentSize=820320, freeSize=1117800608, maxSize=1118620928, heapSize=820320, minSize=1062689856, minFactor=0.95, multiSize=531344928, multiFactor=0.5, singleSize=265672464, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 19:32:22,209 INFO [StoreOpener-534574363-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 19:32:22,214 DEBUG [StoreOpener-534574363-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/table 2018-01-13 19:32:22,214 DEBUG [StoreOpener-534574363-1] util.CommonFSUtils(559): Set storagePolicy=HOT for path=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/table 2018-01-13 19:32:22,216 INFO [StoreOpener-534574363-1] regionserver.HStore(293): Memstore class name is org.apache.hadoop.hbase.regionserver.DefaultMemStore 2018-01-13 19:32:22,216 INFO [StoreOpener-534574363-1] hfile.CacheConfig(239): Created cacheConfig for table: blockCache=LruBlockCache{blockCount=0, currentSize=820320, freeSize=1117800608, maxSize=1118620928, heapSize=820320, minSize=1062689856, minFactor=0.95, multiSize=531344928, multiFactor=0.5, singleSize=265672464, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-01-13 19:32:22,218 INFO [StoreOpener-534574363-1] compactions.CompactionConfiguration(146): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2018-01-13 19:32:22,222 DEBUG [RS_OPEN_META-ubuntu:44139-0] regionserver.FlushLargeStoresPolicy(60): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushSize/# of families (25.6M)) instead. 2018-01-13 19:32:22,223 INFO [RS_OPEN_META-ubuntu:44139-0] regionserver.HRegion(960): Onlined 534574363; next sequenceid=2 2018-01-13 19:32:22,223 INFO [PostOpenDeployTasks:534574363] regionserver.HRegionServer(2197): Post open deploy tasks for hbase:meta,,1_0001.534574363 2018-01-13 19:32:22,224 WARN [ReadOnlyZKClient] client.ZKAsyncRegistry(168): Failed to fetch /hbase/meta-region-server-2 org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/meta-region-server-2 at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:164) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:321) at java.lang.Thread.run(Thread.java:748) 2018-01-13 19:32:22,228 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=36079] assignment.RegionTransitionProcedure(230): Received report OPENED seqId=2, pid=7, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=534574363; rit=OPENING, location=ubuntu,44139,1515843136509 2018-01-13 19:32:22,229 DEBUG [ProcExecWrkr-16] hbase.MetaTableAccessor(2131): Put {"totalColumns":1,"row":"hbase:meta","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":1515843142228}]},"ts":1515843142228} 2018-01-13 19:32:22,229 DEBUG [PostOpenDeployTasks:534574363] regionserver.HRegionServer(2221): Finished post open deploy task for hbase:meta,,1_0001.534574363 2018-01-13 19:32:22,230 DEBUG [RS_OPEN_META-ubuntu:44139-0] handler.OpenRegionHandler(128): Opened hbase:meta,,1_0001.534574363 on ubuntu,44139,1515843136509 2018-01-13 19:32:22,234 INFO [ProcExecWrkr-16] hbase.MetaTableAccessor(1786): Updated table hbase:meta state to ENABLED in META 2018-01-13 19:32:22,234 INFO [ProcExecWrkr-16] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=1) location in ZooKeeper as ubuntu,44139,1515843136509 2018-01-13 19:32:22,237 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server-1 2018-01-13 19:32:22,240 INFO [ProcExecWrkr-16] procedure2.ProcedureExecutor(1250): Finished pid=7, state=SUCCESS; AssignProcedure table=hbase:meta, region=534574363 in 515msec 2018-01-13 19:32:22,288 INFO [M:0;ubuntu:36079] master.MasterMetaBootstrap(149): hbase:meta with replicaId 1, location=ubuntu,44139,1515843136509 2018-01-13 19:32:22,289 DEBUG [M:0;ubuntu:36079] zookeeper.ZKUtil(614): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server-2 because node does not exist (not an error) 2018-01-13 19:32:22,289 DEBUG [M:0;ubuntu:36079] master.MasterMetaBootstrap(130): meta state from zookeeper: {1657623790 state=OFFLINE, ts=1515843142289, server=null} 2018-01-13 19:32:22,289 DEBUG [M:0;ubuntu:36079] assignment.AssignmentManager(467): Assigning hbase:meta,,1_0002.1657623790 2018-01-13 19:32:22,397 DEBUG [M:0;ubuntu:36079] procedure2.ProcedureExecutor(869): Stored pid=8, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1657623790 2018-01-13 19:32:22,400 INFO [ProcExecWrkr-1] procedure.MasterProcedureScheduler(883): pid=8, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1657623790 hbase:meta hbase:meta,,1_0002.1657623790 2018-01-13 19:32:22,400 INFO [ProcExecWrkr-1] assignment.AssignProcedure(211): Start pid=8, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1657623790; rit=OFFLINE, location=null; forceNewPlan=false, retain=false 2018-01-13 19:32:22,428 WARN [ReadOnlyZKClient] client.ZKAsyncRegistry(168): Failed to fetch /hbase/meta-region-server-2 org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/meta-region-server-2 at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:164) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:321) at java.lang.Thread.run(Thread.java:748) 2018-01-13 19:32:22,448 DEBUG [M:0;ubuntu:36079] procedure.ProcedureSyncWait(189): waitFor pid=8 2018-01-13 19:32:22,553 DEBUG [AssignmentThread] assignment.AssignmentManager(1758): Processing assignment plans for System tables sysServersCount=3, allServersCount=3 2018-01-13 19:32:22,554 DEBUG [AssignmentThread] balancer.RegionLocationFinder(279): HDFSBlocksDistribution not found in cache for region hbase:meta,,1_0002.1657623790 2018-01-13 19:32:22,584 INFO [ProcExecWrkr-2] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=2) location in ZooKeeper as ubuntu,40283,1515843136612 2018-01-13 19:32:22,588 DEBUG [ProcExecWrkr-2] zookeeper.MetaTableLocator(468): META region location doesn't exist for replicaId=2, create it 2018-01-13 19:32:22,592 INFO [ProcExecWrkr-2] assignment.RegionTransitionProcedure(208): Dispatch pid=8, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1657623790; rit=OPENING, location=ubuntu,40283,1515843136612 2018-01-13 19:32:22,636 INFO [main] zookeeper.ReadOnlyZKClient(130): Start read only zookeeper connection 0x11e355ca to 127.0.0.1:53209, session timeout 90000 ms, retries 30, retry interval 1000 ms, keep alive 60000 ms 2018-01-13 19:32:22,637 INFO [main] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x11e355ca to 127.0.0.1:53209 2018-01-13 19:32:22,644 INFO [main] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x52d6d273 to localhost:53209 2018-01-13 19:32:22,644 INFO [main] hbase.HBaseTestingUtility(1085): Shutting down minicluster 2018-01-13 19:32:22,655 INFO [main] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x2aa7399c to localhost:53209 2018-01-13 19:32:22,656 DEBUG [main] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 19:32:22,660 DEBUG [main] util.JVMClusterUtil(238): Shutting down HBase Cluster 2018-01-13 19:32:22,660 INFO [main] master.ServerManager(962): Cluster shutdown requested of master=ubuntu,36079,1515843135537 2018-01-13 19:32:22,669 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:40283-0x160ef4aa4c30002, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 19:32:22,670 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:38841-0x160ef4aa4c30003, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 19:32:22,670 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:44139-0x160ef4aa4c30001, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 19:32:22,669 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'ubuntu,44139,1515843136509' ***** 2018-01-13 19:32:22,669 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2018-01-13 19:32:22,670 INFO [main] regionserver.HRegionServer(2172): STOPPED: Shutdown requested 2018-01-13 19:32:22,670 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:38841-0x160ef4aa4c30003, quorum=localhost:53209, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 19:32:22,671 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:40283-0x160ef4aa4c30002, quorum=localhost:53209, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 19:32:22,670 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'ubuntu,40283,1515843136612' ***** 2018-01-13 19:32:22,670 INFO [RS:0;ubuntu:44139] regionserver.SplitLogWorker(160): Sending interrupt to stop the worker thread 2018-01-13 19:32:22,671 INFO [SplitLogWorker-ubuntu:44139] regionserver.SplitLogWorker(142): SplitLogWorker interrupted. Exiting. 2018-01-13 19:32:22,671 INFO [SplitLogWorker-ubuntu:44139] regionserver.SplitLogWorker(151): SplitLogWorker ubuntu,44139,1515843136509 exiting 2018-01-13 19:32:22,671 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 19:32:22,671 DEBUG [main-EventThread] zookeeper.ZKUtil(357): regionserver:44139-0x160ef4aa4c30001, quorum=localhost:53209, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2018-01-13 19:32:22,672 INFO [RS:0;ubuntu:44139] regionserver.HeapMemoryManager(221): Stopping HeapMemoryTuner chore. 2018-01-13 19:32:22,672 INFO [RS:0;ubuntu:44139] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager gracefully. 2018-01-13 19:32:22,672 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.1 exiting 2018-01-13 19:32:22,672 INFO [RS:0;ubuntu:44139] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2018-01-13 19:32:22,678 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.0 exiting 2018-01-13 19:32:22,677 INFO [main] regionserver.HRegionServer(2172): STOPPED: Shutdown requested 2018-01-13 19:32:22,679 INFO [main] regionserver.HRegionServer(2158): ***** STOPPING region server 'ubuntu,38841,1515843136669' ***** 2018-01-13 19:32:22,680 INFO [main] regionserver.HRegionServer(2172): STOPPED: Shutdown requested 2018-01-13 19:32:22,682 INFO [RS:2;ubuntu:38841] regionserver.SplitLogWorker(160): Sending interrupt to stop the worker thread 2018-01-13 19:32:22,679 INFO [RS:1;ubuntu:40283] regionserver.SplitLogWorker(160): Sending interrupt to stop the worker thread 2018-01-13 19:32:22,682 INFO [SplitLogWorker-ubuntu:38841] regionserver.SplitLogWorker(142): SplitLogWorker interrupted. Exiting. 2018-01-13 19:32:22,682 INFO [SplitLogWorker-ubuntu:40283] regionserver.SplitLogWorker(142): SplitLogWorker interrupted. Exiting. 2018-01-13 19:32:22,682 INFO [SplitLogWorker-ubuntu:40283] regionserver.SplitLogWorker(151): SplitLogWorker ubuntu,40283,1515843136612 exiting 2018-01-13 19:32:22,682 INFO [RS:2;ubuntu:38841] regionserver.HeapMemoryManager(221): Stopping HeapMemoryTuner chore. 2018-01-13 19:32:22,682 DEBUG [RS_CLOSE_REGION-ubuntu:44139-0] handler.CloseRegionHandler(92): Processing close of hbase:namespace,,1515843140318.29330989a4082382160ba4b6b47c5e66. 2018-01-13 19:32:22,680 INFO [RS:0;ubuntu:44139] regionserver.HRegionServer(1081): stopping server ubuntu,44139,1515843136509 2018-01-13 19:32:22,685 DEBUG [RS:0;ubuntu:44139] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 19:32:22,685 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.0 exiting 2018-01-13 19:32:22,683 INFO [RS:2;ubuntu:38841] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager gracefully. 2018-01-13 19:32:22,686 INFO [RS:2;ubuntu:38841] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2018-01-13 19:32:22,687 INFO [RS:2;ubuntu:38841] regionserver.HRegionServer(1081): stopping server ubuntu,38841,1515843136669 2018-01-13 19:32:22,687 DEBUG [RS:2;ubuntu:38841] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 19:32:22,687 INFO [RS:2;ubuntu:38841] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x14f00d46 to localhost:53209 2018-01-13 19:32:22,687 DEBUG [RS:2;ubuntu:38841] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 19:32:22,683 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.1 exiting 2018-01-13 19:32:22,687 INFO [RS:2;ubuntu:38841] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-01-13 19:32:22,687 INFO [RS:2;ubuntu:38841] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-01-13 19:32:22,688 INFO [RS:2;ubuntu:38841] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-01-13 19:32:22,682 INFO [SplitLogWorker-ubuntu:38841] regionserver.SplitLogWorker(151): SplitLogWorker ubuntu,38841,1515843136669 exiting 2018-01-13 19:32:22,682 INFO [RS:1;ubuntu:40283] regionserver.HeapMemoryManager(221): Stopping HeapMemoryTuner chore. 2018-01-13 19:32:22,687 DEBUG [RS_CLOSE_REGION-ubuntu:44139-0] regionserver.HRegion(1481): Closing hbase:namespace,,1515843140318.29330989a4082382160ba4b6b47c5e66.: disabling compactions & flushes 2018-01-13 19:32:22,685 INFO [RS:0;ubuntu:44139] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x6f8ca84a to localhost:53209 2018-01-13 19:32:22,688 DEBUG [RS:0;ubuntu:44139] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 19:32:22,688 INFO [RS:0;ubuntu:44139] regionserver.HRegionServer(1388): Waiting on 2 regions to close 2018-01-13 19:32:22,688 DEBUG [RS:0;ubuntu:44139] regionserver.HRegionServer(1392): Online Regions={29330989a4082382160ba4b6b47c5e66=hbase:namespace,,1515843140318.29330989a4082382160ba4b6b47c5e66., 534574363=hbase:meta,,1_0001.534574363} 2018-01-13 19:32:22,689 INFO [RS:2;ubuntu:38841] regionserver.HRegionServer(1388): Waiting on 1 regions to close 2018-01-13 19:32:22,689 DEBUG [RS:2;ubuntu:38841] regionserver.HRegionServer(1392): Online Regions={1588230740=hbase:meta,,1.1588230740} 2018-01-13 19:32:22,689 DEBUG [RS_CLOSE_REGION-ubuntu:44139-0] regionserver.HRegion(1520): Updates disabled for region hbase:namespace,,1515843140318.29330989a4082382160ba4b6b47c5e66. 2018-01-13 19:32:22,690 INFO [RS_CLOSE_REGION-ubuntu:44139-0] regionserver.HRegion(2516): Flushing 1/1 column families, memstore=78 B 2018-01-13 19:32:22,693 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.1 exiting 2018-01-13 19:32:22,694 DEBUG [RS_CLOSE_META-ubuntu:44139-0] handler.CloseRegionHandler(92): Processing close of hbase:meta,,1_0001.534574363 2018-01-13 19:32:22,694 DEBUG [RS_CLOSE_META-ubuntu:38841-0] handler.CloseRegionHandler(92): Processing close of hbase:meta,,1.1588230740 2018-01-13 19:32:22,694 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(273): MemStoreFlusher.0 exiting 2018-01-13 19:32:22,703 DEBUG [RS_CLOSE_META-ubuntu:38841-0] regionserver.HRegion(1481): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2018-01-13 19:32:22,696 INFO [RS:1;ubuntu:40283] flush.RegionServerFlushTableProcedureManager(116): Stopping region server flush procedure manager gracefully. 2018-01-13 19:32:22,703 INFO [RS:1;ubuntu:40283] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2018-01-13 19:32:22,696 DEBUG [RS_CLOSE_META-ubuntu:44139-0] regionserver.HRegion(1481): Closing hbase:meta,,1_0001.534574363: disabling compactions & flushes 2018-01-13 19:32:22,703 INFO [RS:1;ubuntu:40283] regionserver.HRegionServer(1081): stopping server ubuntu,40283,1515843136612 2018-01-13 19:32:22,703 DEBUG [RS:1;ubuntu:40283] zookeeper.MetaTableLocator(641): Stopping MetaTableLocator 2018-01-13 19:32:22,703 DEBUG [RS_CLOSE_META-ubuntu:38841-0] regionserver.HRegion(1520): Updates disabled for region hbase:meta,,1.1588230740 2018-01-13 19:32:22,703 INFO [RS:1;ubuntu:40283] zookeeper.ReadOnlyZKClient(342): Close zookeeper connection 0x4e704e39 to localhost:53209 2018-01-13 19:32:22,704 DEBUG [RS:1;ubuntu:40283] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 19:32:22,704 INFO [RS:1;ubuntu:40283] regionserver.HRegionServer(1109): stopping server ubuntu,40283,1515843136612; all regions closed. 2018-01-13 19:32:22,704 INFO [RS_CLOSE_META-ubuntu:38841-0] regionserver.HRegion(2516): Flushing 5/5 column families, memstore=1.30 KB 2018-01-13 19:32:22,705 DEBUG [RS_CLOSE_META-ubuntu:44139-0] regionserver.HRegion(1520): Updates disabled for region hbase:meta,,1_0001.534574363 2018-01-13 19:32:22,709 INFO [StoreCloserThread-hbase:meta,,1_0001.534574363-1] regionserver.HStore(930): Closed info 2018-01-13 19:32:22,710 INFO [StoreCloserThread-hbase:meta,,1_0001.534574363-1] regionserver.HStore(930): Closed rep_barrier 2018-01-13 19:32:22,710 INFO [StoreCloserThread-hbase:meta,,1_0001.534574363-1] regionserver.HStore(930): Closed rep_meta 2018-01-13 19:32:22,710 INFO [StoreCloserThread-hbase:meta,,1_0001.534574363-1] regionserver.HStore(930): Closed rep_position 2018-01-13 19:32:22,710 INFO [StoreCloserThread-hbase:meta,,1_0001.534574363-1] regionserver.HStore(930): Closed table 2018-01-13 19:32:22,710 DEBUG [RS_CLOSE_META-ubuntu:44139-0] coprocessor.CoprocessorHost(289): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-01-13 19:32:22,719 INFO [RS_CLOSE_META-ubuntu:44139-0] regionserver.HRegion(1633): Closed hbase:meta,,1_0001.534574363 2018-01-13 19:32:22,719 DEBUG [RS_CLOSE_META-ubuntu:44139-0] handler.CloseRegionHandler(125): Closed hbase:meta,,1_0001.534574363 2018-01-13 19:32:22,720 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35477 is added to blk_1073741831_1007{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c786ffef-0cfe-40db-9609-e37ce51fcb98:NORMAL:127.0.0.1:39981|RBW], ReplicaUC[[DISK]DS-2ae3c181-7033-4bb1-926b-79afb8ba9668:NORMAL:127.0.0.1:45841|RBW], ReplicaUC[[DISK]DS-3a12d4a8-59eb-4588-98a1-23de755fb453:NORMAL:127.0.0.1:35477|RBW]]} size 0 2018-01-13 19:32:22,727 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45841 is added to blk_1073741831_1007{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c786ffef-0cfe-40db-9609-e37ce51fcb98:NORMAL:127.0.0.1:39981|RBW], ReplicaUC[[DISK]DS-2ae3c181-7033-4bb1-926b-79afb8ba9668:NORMAL:127.0.0.1:45841|RBW], ReplicaUC[[DISK]DS-3a12d4a8-59eb-4588-98a1-23de755fb453:NORMAL:127.0.0.1:35477|RBW]]} size 0 2018-01-13 19:32:22,728 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39981 is added to blk_1073741831_1007{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c786ffef-0cfe-40db-9609-e37ce51fcb98:NORMAL:127.0.0.1:39981|RBW], ReplicaUC[[DISK]DS-2ae3c181-7033-4bb1-926b-79afb8ba9668:NORMAL:127.0.0.1:45841|RBW], ReplicaUC[[DISK]DS-3a12d4a8-59eb-4588-98a1-23de755fb453:NORMAL:127.0.0.1:35477|RBW]]} size 0 2018-01-13 19:32:22,744 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=ubuntu,40283,1515843136612 version=2097152 2018-01-13 19:32:22,746 DEBUG [RSProcedureDispatcher-pool4-t4] master.ServerManager(783): New admin connection to ubuntu,40283,1515843136612 2018-01-13 19:32:22,759 DEBUG [RSProcedureDispatcher-pool4-t4] ipc.NettyRpcConnection(255): Connecting to ubuntu/127.0.1.1:40283 2018-01-13 19:32:22,759 DEBUG [RS:1;ubuntu:40283] wal.AbstractFSWAL(861): Moved 1 WAL file(s) to /user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/oldWALs 2018-01-13 19:32:22,769 INFO [RS:1;ubuntu:40283] wal.AbstractFSWAL(864): Closed WAL: AsyncFSWAL ubuntu%2C40283%2C1515843136612:(num 1515843139217) 2018-01-13 19:32:22,769 DEBUG [RS:1;ubuntu:40283] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 19:32:22,770 INFO [RS:1;ubuntu:40283] regionserver.Leases(147): RS:1;ubuntu:40283 closing leases 2018-01-13 19:32:22,770 INFO [RS:1;ubuntu:40283] regionserver.Leases(150): RS:1;ubuntu:40283 closed leases 2018-01-13 19:32:22,770 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(527): Auth successful for zhangduo (auth:SIMPLE) 2018-01-13 19:32:22,774 INFO [RS:1;ubuntu:40283] hbase.ChoreService(327): Chore service for: ubuntu,40283,1515843136612 had [[ScheduledChore: Name: MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region ubuntu,40283,1515843136612 Period: 120000 Unit: MILLISECONDS]] on shutdown 2018-01-13 19:32:22,776 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(557): Connection from 127.0.0.1 port: 35060 with version info: version: "2.0.0-beta-1" url: "git://ubuntu/home/zhangduo/hbase/code" revision: "a90cf1bc134a14ba7b692c8eb6ee7cddc4b857bf" user: "zhangduo" date: "Fri Jan 12 21:37:44 CST 2018" src_checksum: "5c5be6fe364dffb36035e82ded054eaa" version_major: 2 version_minor: 0 2018-01-13 19:32:22,776 INFO [regionserver/ubuntu/127.0.1.1:0.logRoller] regionserver.LogRoller(195): LogRoller exiting. 2018-01-13 19:32:22,776 INFO [RS:1;ubuntu:40283] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-01-13 19:32:22,783 INFO [RS:1;ubuntu:40283] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-01-13 19:32:22,783 INFO [RS:1;ubuntu:40283] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-01-13 19:32:22,784 INFO [RS:1;ubuntu:40283] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 19:32:22,785 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=5,queue=0,port=40283] ipc.CallRunner(141): callId: 29 service: AdminService methodName: ExecuteProcedures size: 78 connection: 127.0.0.1:35060 deadline: 1515843202784 org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server ubuntu,40283,1515843136612 stopping at org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1418) at org.apache.hadoop.hbase.regionserver.RSRpcServices.executeProcedures(RSRpcServices.java:3461) at org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:28581) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:404) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) 2018-01-13 19:32:22,791 WARN [RSProcedureDispatcher-pool4-t4] procedure.RSProcedureDispatcher$AbstractRSRemoteCall(204): Failed dispatch to server=ubuntu,40283,1515843136612 try=0 java.io.IOException: Call to ubuntu/127.0.1.1:40283 failed on local exception: java.io.IOException: Connection closed at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$3(AbstractRpcClient.java:375) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:1) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.cleanupCalls(NettyRpcDuplexHandler.java:202) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelInactive(NettyRpcDuplexHandler.java:210) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:377) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1354) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:917) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:309) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Connection closed ... 25 more 2018-01-13 19:32:22,798 WARN [RSProcedureDispatcher-pool4-t4] assignment.RegionTransitionProcedure(182): Remote call failed pid=8, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1657623790; rit=OPENING, location=ubuntu,40283,1515843136612; exception=Call to ubuntu/127.0.1.1:40283 failed on local exception: java.io.IOException: Connection closed 2018-01-13 19:32:22,798 INFO [RSProcedureDispatcher-pool4-t4] assignment.AssignProcedure(330): Retry=1 of max=10; pid=8, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1657623790; rit=OPENING, location=ubuntu,40283,1515843136612 2018-01-13 19:32:22,800 INFO [ProcExecWrkr-3] assignment.AssignProcedure(211): Start pid=8, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1657623790; rit=OFFLINE, location=null; forceNewPlan=true, retain=false 2018-01-13 19:32:22,801 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ubuntu,40283,1515843136612 2018-01-13 19:32:22,801 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:38841-0x160ef4aa4c30003, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ubuntu,40283,1515843136612 2018-01-13 19:32:22,801 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:44139-0x160ef4aa4c30001, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ubuntu,40283,1515843136612 2018-01-13 19:32:22,801 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:44139-0x160ef4aa4c30001, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 19:32:22,801 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:38841-0x160ef4aa4c30003, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 19:32:22,801 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:40283-0x160ef4aa4c30002, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ubuntu,40283,1515843136612 2018-01-13 19:32:22,801 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:40283-0x160ef4aa4c30002, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 19:32:22,801 INFO [main-EventThread] master.RegionServerTracker(122): RegionServer ephemeral node deleted, processing expiration [ubuntu,40283,1515843136612] 2018-01-13 19:32:22,804 INFO [main-EventThread] master.ServerManager(593): Cluster shutdown set; ubuntu,40283,1515843136612 expired; onlineServers=2 2018-01-13 19:32:22,805 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 19:32:22,807 INFO [RS:1;ubuntu:40283] regionserver.HRegionServer(1155): stopping server ubuntu,40283,1515843136612; zookeeper connection closed. 2018-01-13 19:32:22,807 INFO [RS:1;ubuntu:40283] regionserver.HRegionServer(1157): RS:1;ubuntu:40283 exiting 2018-01-13 19:32:22,807 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@6b5dd0b6] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(209): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@6b5dd0b6 2018-01-13 19:32:22,810 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,38841,1515843136669 2018-01-13 19:32:22,812 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,44139,1515843136509 2018-01-13 19:32:22,860 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45841 is added to blk_1073741836_1012{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2682b908-d71d-4edd-a7d6-c00e22c135f0:NORMAL:127.0.0.1:39981|RBW], ReplicaUC[[DISK]DS-3a12d4a8-59eb-4588-98a1-23de755fb453:NORMAL:127.0.0.1:35477|RBW], ReplicaUC[[DISK]DS-6e234590-c709-4272-a78c-d0a2418a7171:NORMAL:127.0.0.1:45841|FINALIZED]]} size 0 2018-01-13 19:32:22,860 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35477 is added to blk_1073741836_1012{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2682b908-d71d-4edd-a7d6-c00e22c135f0:NORMAL:127.0.0.1:39981|RBW], ReplicaUC[[DISK]DS-3a12d4a8-59eb-4588-98a1-23de755fb453:NORMAL:127.0.0.1:35477|RBW], ReplicaUC[[DISK]DS-6e234590-c709-4272-a78c-d0a2418a7171:NORMAL:127.0.0.1:45841|FINALIZED]]} size 0 2018-01-13 19:32:22,862 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39981 is added to blk_1073741836_1012{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2682b908-d71d-4edd-a7d6-c00e22c135f0:NORMAL:127.0.0.1:39981|RBW], ReplicaUC[[DISK]DS-3a12d4a8-59eb-4588-98a1-23de755fb453:NORMAL:127.0.0.1:35477|RBW], ReplicaUC[[DISK]DS-6e234590-c709-4272-a78c-d0a2418a7171:NORMAL:127.0.0.1:45841|FINALIZED]]} size 0 2018-01-13 19:32:22,863 INFO [RS_CLOSE_META-ubuntu:38841-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=12, memsize=1.1 K, hasBloomFilter=false, into tmp file hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/.tmp/info/c9b84605337a4a5db60171537911c739 2018-01-13 19:32:22,864 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39981 is added to blk_1073741837_1013{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2ae3c181-7033-4bb1-926b-79afb8ba9668:NORMAL:127.0.0.1:45841|RBW], ReplicaUC[[DISK]DS-7b74f716-787f-414f-bbbf-7d604b248220:NORMAL:127.0.0.1:35477|RBW], ReplicaUC[[DISK]DS-c786ffef-0cfe-40db-9609-e37ce51fcb98:NORMAL:127.0.0.1:39981|RBW]]} size 0 2018-01-13 19:32:22,869 INFO [RS_CLOSE_REGION-ubuntu:44139-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=6, memsize=78, hasBloomFilter=true, into tmp file hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/namespace/29330989a4082382160ba4b6b47c5e66/.tmp/info/9cc85f6d02fe431d9c5ac8a64ef770d9 2018-01-13 19:32:22,870 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35477 is added to blk_1073741837_1013 size 4898 2018-01-13 19:32:22,871 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45841 is added to blk_1073741837_1013 size 4898 2018-01-13 19:32:22,952 DEBUG [AssignmentThread] assignment.AssignmentManager(1758): Processing assignment plans for System tables sysServersCount=2, allServersCount=2 2018-01-13 19:32:22,954 INFO [ProcExecWrkr-4] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=2) location in ZooKeeper as ubuntu,44139,1515843136509 2018-01-13 19:32:22,955 DEBUG [RS_CLOSE_REGION-ubuntu:44139-0] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/namespace/29330989a4082382160ba4b6b47c5e66/.tmp/info/9cc85f6d02fe431d9c5ac8a64ef770d9 as hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/namespace/29330989a4082382160ba4b6b47c5e66/info/9cc85f6d02fe431d9c5ac8a64ef770d9 2018-01-13 19:32:22,957 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server-2 2018-01-13 19:32:22,957 INFO [ProcExecWrkr-4] assignment.RegionTransitionProcedure(208): Dispatch pid=8, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1657623790; rit=OPENING, location=ubuntu,44139,1515843136509 2018-01-13 19:32:22,969 INFO [RS_CLOSE_REGION-ubuntu:44139-0] regionserver.HStore(1028): Added hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/namespace/29330989a4082382160ba4b6b47c5e66/info/9cc85f6d02fe431d9c5ac8a64ef770d9, entries=2, sequenceid=6, filesize=4.8 K 2018-01-13 19:32:22,978 INFO [RS_CLOSE_REGION-ubuntu:44139-0] regionserver.HRegion(2712): Finished memstore flush of ~78 B/78, currentsize=0 B/0 for region hbase:namespace,,1515843140318.29330989a4082382160ba4b6b47c5e66. in 288ms, sequenceid=6, compaction requested=false 2018-01-13 19:32:22,990 INFO [regionserver/ubuntu/127.0.1.1:0.leaseChecker] regionserver.Leases(147): regionserver/ubuntu/127.0.1.1:0.leaseChecker closing leases 2018-01-13 19:32:22,990 INFO [regionserver/ubuntu/127.0.1.1:0.leaseChecker] regionserver.Leases(150): regionserver/ubuntu/127.0.1.1:0.leaseChecker closed leases 2018-01-13 19:32:22,992 INFO [StoreCloserThread-hbase:namespace,,1515843140318.29330989a4082382160ba4b6b47c5e66.-1] regionserver.HStore(930): Closed info 2018-01-13 19:32:23,002 INFO [regionserver/ubuntu/127.0.1.1:0.leaseChecker] regionserver.Leases(147): regionserver/ubuntu/127.0.1.1:0.leaseChecker closing leases 2018-01-13 19:32:23,002 INFO [regionserver/ubuntu/127.0.1.1:0.leaseChecker] regionserver.Leases(150): regionserver/ubuntu/127.0.1.1:0.leaseChecker closed leases 2018-01-13 19:32:23,011 DEBUG [RS_CLOSE_REGION-ubuntu:44139-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/namespace/29330989a4082382160ba4b6b47c5e66/recovered.edits/9.seqid, newSeqId=9, maxSeqId=2 2018-01-13 19:32:23,012 INFO [RS_CLOSE_REGION-ubuntu:44139-0] regionserver.HRegion(1633): Closed hbase:namespace,,1515843140318.29330989a4082382160ba4b6b47c5e66. 2018-01-13 19:32:23,012 DEBUG [RS_CLOSE_REGION-ubuntu:44139-0] handler.CloseRegionHandler(125): Closed hbase:namespace,,1515843140318.29330989a4082382160ba4b6b47c5e66. 2018-01-13 19:32:23,013 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45841 is added to blk_1073741838_1014{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c786ffef-0cfe-40db-9609-e37ce51fcb98:NORMAL:127.0.0.1:39981|RBW], ReplicaUC[[DISK]DS-3a12d4a8-59eb-4588-98a1-23de755fb453:NORMAL:127.0.0.1:35477|RBW], ReplicaUC[[DISK]DS-2ae3c181-7033-4bb1-926b-79afb8ba9668:NORMAL:127.0.0.1:45841|RBW]]} size 0 2018-01-13 19:32:23,014 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35477 is added to blk_1073741838_1014{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c786ffef-0cfe-40db-9609-e37ce51fcb98:NORMAL:127.0.0.1:39981|RBW], ReplicaUC[[DISK]DS-3a12d4a8-59eb-4588-98a1-23de755fb453:NORMAL:127.0.0.1:35477|RBW], ReplicaUC[[DISK]DS-2ae3c181-7033-4bb1-926b-79afb8ba9668:NORMAL:127.0.0.1:45841|RBW]]} size 0 2018-01-13 19:32:23,020 INFO [regionserver/ubuntu/127.0.1.1:0.leaseChecker] regionserver.Leases(147): regionserver/ubuntu/127.0.1.1:0.leaseChecker closing leases 2018-01-13 19:32:23,021 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39981 is added to blk_1073741838_1014 size 4833 2018-01-13 19:32:23,021 INFO [regionserver/ubuntu/127.0.1.1:0.leaseChecker] regionserver.Leases(150): regionserver/ubuntu/127.0.1.1:0.leaseChecker closed leases 2018-01-13 19:32:23,022 INFO [RS_CLOSE_META-ubuntu:38841-0] regionserver.DefaultStoreFlusher(87): Flushed, sequenceid=12, memsize=178, hasBloomFilter=false, into tmp file hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/.tmp/table/5fcd7a823ca84e2b8aaa8ebbbffadec4 2018-01-13 19:32:23,033 DEBUG [RS_CLOSE_META-ubuntu:38841-0] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/.tmp/info/c9b84605337a4a5db60171537911c739 as hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/info/c9b84605337a4a5db60171537911c739 2018-01-13 19:32:23,040 INFO [ubuntu,38841,1515843136669_Chore_1] hbase.ScheduledChore(180): Chore: MemstoreFlusherChore was stopped 2018-01-13 19:32:23,045 INFO [RS_CLOSE_META-ubuntu:38841-0] regionserver.HStore(1028): Added hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/info/c9b84605337a4a5db60171537911c739, entries=10, sequenceid=12, filesize=5.8 K 2018-01-13 19:32:23,050 DEBUG [RS_CLOSE_META-ubuntu:38841-0] regionserver.HRegionFileSystem(463): Committing store file hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/.tmp/table/5fcd7a823ca84e2b8aaa8ebbbffadec4 as hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/table/5fcd7a823ca84e2b8aaa8ebbbffadec4 2018-01-13 19:32:23,067 INFO [RS_CLOSE_META-ubuntu:38841-0] regionserver.HStore(1028): Added hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/table/5fcd7a823ca84e2b8aaa8ebbbffadec4, entries=4, sequenceid=12, filesize=4.7 K 2018-01-13 19:32:23,069 INFO [RS_CLOSE_META-ubuntu:38841-0] regionserver.HRegion(2712): Finished memstore flush of ~1.30 KB/1336, currentsize=0 B/0 for region hbase:meta,,1.1588230740 in 365ms, sequenceid=12, compaction requested=false 2018-01-13 19:32:23,071 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed info 2018-01-13 19:32:23,073 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_barrier 2018-01-13 19:32:23,073 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_meta 2018-01-13 19:32:23,074 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed rep_position 2018-01-13 19:32:23,081 INFO [ubuntu,44139,1515843136509_Chore_1] hbase.ScheduledChore(180): Chore: MemstoreFlusherChore was stopped 2018-01-13 19:32:23,083 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(930): Closed table 2018-01-13 19:32:23,093 INFO [RS:0;ubuntu:44139] regionserver.HRegionServer(1109): stopping server ubuntu,44139,1515843136509; all regions closed. 2018-01-13 19:32:23,102 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39981 is added to blk_1073741830_1006{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2682b908-d71d-4edd-a7d6-c00e22c135f0:NORMAL:127.0.0.1:39981|RBW], ReplicaUC[[DISK]DS-2ae3c181-7033-4bb1-926b-79afb8ba9668:NORMAL:127.0.0.1:45841|RBW], ReplicaUC[[DISK]DS-7b74f716-787f-414f-bbbf-7d604b248220:NORMAL:127.0.0.1:35477|RBW]]} size 0 2018-01-13 19:32:23,102 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35477 is added to blk_1073741830_1006{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2682b908-d71d-4edd-a7d6-c00e22c135f0:NORMAL:127.0.0.1:39981|RBW], ReplicaUC[[DISK]DS-2ae3c181-7033-4bb1-926b-79afb8ba9668:NORMAL:127.0.0.1:45841|RBW], ReplicaUC[[DISK]DS-7b74f716-787f-414f-bbbf-7d604b248220:NORMAL:127.0.0.1:35477|RBW]]} size 0 2018-01-13 19:32:23,102 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45841 is added to blk_1073741830_1006{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-2682b908-d71d-4edd-a7d6-c00e22c135f0:NORMAL:127.0.0.1:39981|RBW], ReplicaUC[[DISK]DS-2ae3c181-7033-4bb1-926b-79afb8ba9668:NORMAL:127.0.0.1:45841|RBW], ReplicaUC[[DISK]DS-7b74f716-787f-414f-bbbf-7d604b248220:NORMAL:127.0.0.1:35477|RBW]]} size 0 2018-01-13 19:32:23,105 DEBUG [RS_CLOSE_META-ubuntu:38841-0] wal.WALSplitter(675): Wrote file=hdfs://localhost:44545/user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/data/hbase/meta/1588230740/recovered.edits/15.seqid, newSeqId=15, maxSeqId=3 2018-01-13 19:32:23,111 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=ubuntu,44139,1515843136509 version=2097152 2018-01-13 19:32:23,113 DEBUG [RS_CLOSE_META-ubuntu:38841-0] coprocessor.CoprocessorHost(289): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2018-01-13 19:32:23,114 INFO [RS_CLOSE_META-ubuntu:38841-0] regionserver.HRegion(1633): Closed hbase:meta,,1.1588230740 2018-01-13 19:32:23,115 DEBUG [RS_CLOSE_META-ubuntu:38841-0] handler.CloseRegionHandler(125): Closed hbase:meta,,1.1588230740 2018-01-13 19:32:23,115 DEBUG [RS:0;ubuntu:44139] wal.AbstractFSWAL(861): Moved 1 WAL file(s) to /user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/oldWALs 2018-01-13 19:32:23,115 INFO [RS:0;ubuntu:44139] wal.AbstractFSWAL(864): Closed WAL: AsyncFSWAL ubuntu%2C44139%2C1515843136509:(num 1515843139217) 2018-01-13 19:32:23,115 DEBUG [RS:0;ubuntu:44139] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 19:32:23,115 INFO [RS:0;ubuntu:44139] regionserver.Leases(147): RS:0;ubuntu:44139 closing leases 2018-01-13 19:32:23,115 INFO [RS:0;ubuntu:44139] regionserver.Leases(150): RS:0;ubuntu:44139 closed leases 2018-01-13 19:32:23,118 INFO [RS:0;ubuntu:44139] hbase.ChoreService(327): Chore service for: ubuntu,44139,1515843136509 had [[ScheduledChore: Name: MovedRegionsCleaner for region ubuntu,44139,1515843136509 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS]] on shutdown 2018-01-13 19:32:23,119 INFO [RS:0;ubuntu:44139] regionserver.CompactSplit(394): Waiting for Split Thread to finish... 2018-01-13 19:32:23,120 INFO [RS:0;ubuntu:44139] regionserver.CompactSplit(394): Waiting for Large Compaction Thread to finish... 2018-01-13 19:32:23,120 INFO [RS:0;ubuntu:44139] regionserver.CompactSplit(394): Waiting for Small Compaction Thread to finish... 2018-01-13 19:32:23,120 INFO [RS:0;ubuntu:44139] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 19:32:23,119 INFO [regionserver/ubuntu/127.0.1.1:0.logRoller] regionserver.LogRoller(195): LogRoller exiting. 2018-01-13 19:32:23,129 WARN [RSProcedureDispatcher-pool4-t5] procedure.RSProcedureDispatcher$AbstractRSRemoteCall(204): Failed dispatch to server=ubuntu,44139,1515843136509 try=0 java.io.IOException: Call to ubuntu/127.0.1.1:44139 failed on local exception: java.io.IOException: Connection closed at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$3(AbstractRpcClient.java:375) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:1) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.cleanupCalls(NettyRpcDuplexHandler.java:202) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelInactive(NettyRpcDuplexHandler.java:210) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:377) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1354) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:917) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822) at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:309) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Connection closed ... 25 more 2018-01-13 19:32:23,134 WARN [RSProcedureDispatcher-pool4-t5] assignment.RegionTransitionProcedure(182): Remote call failed pid=8, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1657623790; rit=OPENING, location=ubuntu,44139,1515843136509; exception=Call to ubuntu/127.0.1.1:44139 failed on local exception: java.io.IOException: Connection closed 2018-01-13 19:32:23,134 INFO [RSProcedureDispatcher-pool4-t5] assignment.AssignProcedure(330): Retry=2 of max=10; pid=8, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1657623790; rit=OPENING, location=ubuntu,44139,1515843136509 2018-01-13 19:32:23,134 INFO [ProcExecWrkr-5] assignment.AssignProcedure(211): Start pid=8, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1657623790; rit=OFFLINE, location=null; forceNewPlan=true, retain=false 2018-01-13 19:32:23,134 INFO [RS:0;ubuntu:44139] regionserver.HRegionServer(1155): stopping server ubuntu,44139,1515843136509; zookeeper connection closed. 2018-01-13 19:32:23,134 INFO [RS:0;ubuntu:44139] regionserver.HRegionServer(1157): RS:0;ubuntu:44139 exiting 2018-01-13 19:32:23,130 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ubuntu,44139,1515843136509 2018-01-13 19:32:23,134 INFO [main-EventThread] master.RegionServerTracker(122): RegionServer ephemeral node deleted, processing expiration [ubuntu,44139,1515843136509] 2018-01-13 19:32:23,130 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:38841-0x160ef4aa4c30003, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ubuntu,44139,1515843136509 2018-01-13 19:32:23,130 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:44139-0x160ef4aa4c30001, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ubuntu,44139,1515843136509 2018-01-13 19:32:23,135 INFO [main-EventThread] master.ServerManager(593): Cluster shutdown set; ubuntu,44139,1515843136509 expired; onlineServers=1 2018-01-13 19:32:23,135 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 19:32:23,139 DEBUG [main-EventThread] zookeeper.ZKUtil(355): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/ubuntu,38841,1515843136669 2018-01-13 19:32:23,139 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@13712f80] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(209): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@13712f80 2018-01-13 19:32:23,285 DEBUG [AssignmentThread] assignment.AssignmentManager(1758): Processing assignment plans for System tables sysServersCount=1, allServersCount=1 2018-01-13 19:32:23,285 INFO [ProcExecWrkr-6] zookeeper.MetaTableLocator(452): Setting hbase:meta (replicaId=2) location in ZooKeeper as ubuntu,38841,1515843136669 2018-01-13 19:32:23,288 INFO [ProcExecWrkr-6] assignment.RegionTransitionProcedure(208): Dispatch pid=8, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1657623790; rit=OPENING, location=ubuntu,38841,1515843136669 2018-01-13 19:32:23,291 INFO [RS:2;ubuntu:38841] regionserver.HRegionServer(1109): stopping server ubuntu,38841,1515843136669; all regions closed. 2018-01-13 19:32:23,303 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45841 is added to blk_1073741833_1009{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c786ffef-0cfe-40db-9609-e37ce51fcb98:NORMAL:127.0.0.1:39981|RBW], ReplicaUC[[DISK]DS-7b74f716-787f-414f-bbbf-7d604b248220:NORMAL:127.0.0.1:35477|RBW], ReplicaUC[[DISK]DS-6e234590-c709-4272-a78c-d0a2418a7171:NORMAL:127.0.0.1:45841|RBW]]} size 0 2018-01-13 19:32:23,304 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35477 is added to blk_1073741833_1009{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c786ffef-0cfe-40db-9609-e37ce51fcb98:NORMAL:127.0.0.1:39981|RBW], ReplicaUC[[DISK]DS-7b74f716-787f-414f-bbbf-7d604b248220:NORMAL:127.0.0.1:35477|RBW], ReplicaUC[[DISK]DS-6e234590-c709-4272-a78c-d0a2418a7171:NORMAL:127.0.0.1:45841|RBW]]} size 0 2018-01-13 19:32:23,305 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39981 is added to blk_1073741833_1009{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c786ffef-0cfe-40db-9609-e37ce51fcb98:NORMAL:127.0.0.1:39981|RBW], ReplicaUC[[DISK]DS-7b74f716-787f-414f-bbbf-7d604b248220:NORMAL:127.0.0.1:35477|RBW], ReplicaUC[[DISK]DS-6e234590-c709-4272-a78c-d0a2418a7171:NORMAL:127.0.0.1:45841|RBW]]} size 0 2018-01-13 19:32:23,315 DEBUG [RS:2;ubuntu:38841] wal.AbstractFSWAL(861): Moved 1 WAL file(s) to /user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/oldWALs 2018-01-13 19:32:23,316 INFO [RS:2;ubuntu:38841] wal.AbstractFSWAL(864): Closed WAL: AsyncFSWAL ubuntu%2C38841%2C1515843136669.meta:.meta(num 1515843139688) 2018-01-13 19:32:23,323 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35477 is added to blk_1073741832_1008{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-6e234590-c709-4272-a78c-d0a2418a7171:NORMAL:127.0.0.1:45841|RBW], ReplicaUC[[DISK]DS-2682b908-d71d-4edd-a7d6-c00e22c135f0:NORMAL:127.0.0.1:39981|RBW], ReplicaUC[[DISK]DS-3a12d4a8-59eb-4588-98a1-23de755fb453:NORMAL:127.0.0.1:35477|RBW]]} size 0 2018-01-13 19:32:23,324 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39981 is added to blk_1073741832_1008{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-6e234590-c709-4272-a78c-d0a2418a7171:NORMAL:127.0.0.1:45841|RBW], ReplicaUC[[DISK]DS-2682b908-d71d-4edd-a7d6-c00e22c135f0:NORMAL:127.0.0.1:39981|RBW], ReplicaUC[[DISK]DS-3a12d4a8-59eb-4588-98a1-23de755fb453:NORMAL:127.0.0.1:35477|RBW]]} size 0 2018-01-13 19:32:23,326 INFO [Block report processor] blockmanagement.BlockManager(2648): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:45841 is added to blk_1073741832_1008{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-6e234590-c709-4272-a78c-d0a2418a7171:NORMAL:127.0.0.1:45841|RBW], ReplicaUC[[DISK]DS-2682b908-d71d-4edd-a7d6-c00e22c135f0:NORMAL:127.0.0.1:39981|RBW], ReplicaUC[[DISK]DS-3a12d4a8-59eb-4588-98a1-23de755fb453:NORMAL:127.0.0.1:35477|RBW]]} size 0 2018-01-13 19:32:23,332 DEBUG [RS:2;ubuntu:38841] wal.AbstractFSWAL(861): Moved 1 WAL file(s) to /user/zhangduo/test-data/a6b06e3b-00bb-48cb-abac-a6c0980e6433/oldWALs 2018-01-13 19:32:23,333 INFO [RS:2;ubuntu:38841] wal.AbstractFSWAL(864): Closed WAL: AsyncFSWAL ubuntu%2C38841%2C1515843136669:(num 1515843139217) 2018-01-13 19:32:23,333 DEBUG [RS:2;ubuntu:38841] ipc.AbstractRpcClient(483): Stopping rpc client 2018-01-13 19:32:23,333 INFO [RS:2;ubuntu:38841] regionserver.Leases(147): RS:2;ubuntu:38841 closing leases 2018-01-13 19:32:23,334 INFO [RS:2;ubuntu:38841] regionserver.Leases(150): RS:2;ubuntu:38841 closed leases 2018-01-13 19:32:23,336 INFO [RS:2;ubuntu:38841] hbase.ChoreService(327): Chore service for: ubuntu,38841,1515843136669 had [[ScheduledChore: Name: MovedRegionsCleaner for region ubuntu,38841,1515843136669 Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS]] on shutdown 2018-01-13 19:32:23,336 INFO [regionserver/ubuntu/127.0.1.1:0.logRoller] regionserver.LogRoller(195): LogRoller exiting. 2018-01-13 19:32:23,337 INFO [RS:2;ubuntu:38841] ipc.NettyRpcServer(143): Stopping server on 0 2018-01-13 19:32:23,345 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ubuntu,38841,1515843136669 2018-01-13 19:32:23,345 INFO [main-EventThread] master.RegionServerTracker(122): RegionServer ephemeral node deleted, processing expiration [ubuntu,38841,1515843136669] 2018-01-13 19:32:23,345 INFO [main-EventThread] master.ServerManager(593): Cluster shutdown set; ubuntu,38841,1515843136669 expired; onlineServers=0 2018-01-13 19:32:23,345 INFO [main-EventThread] regionserver.HRegionServer(2158): ***** STOPPING region server 'ubuntu,36079,1515843135537' ***** 2018-01-13 19:32:23,345 INFO [main-EventThread] regionserver.HRegionServer(2172): STOPPED: Cluster shutdown set; onlineServer=0 2018-01-13 19:32:23,346 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:38841-0x160ef4aa4c30003, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/ubuntu,38841,1515843136669 2018-01-13 19:32:23,349 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2018-01-13 19:32:23,349 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 19:32:23,350 DEBUG [main-EventThread] zookeeper.ZKWatcher(460): regionserver:38841-0x160ef4aa4c30003, quorum=localhost:53209, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2018-01-13 19:32:23,351 DEBUG [main-EventThread] zookeeper.ZKUtil(357): master:36079-0x160ef4aa4c30000, quorum=localhost:53209, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2018-01-13 19:32:23,353 WARN [main-EventThread] zookeeper.ZKUtil(361): regionserver:38841-0x160ef4aa4c30003, quorum=localhost:53209, baseZNode=/hbase Unable to set watcher on znode /hbase/master org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:206) at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:352) at org.apache.hadoop.hbase.zookeeper.ZKNodeTracker.nodeDeleted(ZKNodeTracker.java:215) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:484) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-01-13 19:32:23,353 ERROR [main-EventThread] zookeeper.ZKWatcher(578): regionserver:38841-0x160ef4aa4c30003, quorum=localhost:53209, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:206) at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:352) at org.apache.hadoop.hbase.zookeeper.ZKNodeTracker.nodeDeleted(ZKNodeTracker.java:215) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:484) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-01-13 19:32:23,358 INFO [RS:2;ubuntu:38841] regionserver.HRegionServer(1155): stopping server ubuntu,38841,1515843136669; zookeeper connection closed. 2018-01-13 19:32:23,358 INFO [RS:2;ubuntu:38841] regionserver.HRegionServer(1157): RS:2;ubuntu:38841 exiting 2018-01-13 19:32:23,359 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@458e3b3f] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(209): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@458e3b3f 2018-01-13 19:32:23,360 INFO [main] util.JVMClusterUtil(322): Shutdown of 1 master(s) and 3 regionserver(s) complete 2018-01-13 19:32:23,360 ERROR [main-EventThread] helpers.MarkerIgnoringBase(159): ***** ABORTING region server ubuntu,38841,1515843136669: Unexpected exception handling nodeDeleted event ***** org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:206) at org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:352) at org.apache.hadoop.hbase.zookeeper.ZKNodeTracker.nodeDeleted(ZKNodeTracker.java:215) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:484) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2018-01-13 19:32:23,361 ERROR [main-EventThread] helpers.MarkerIgnoringBase(143): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2018-01-13 19:32:23,368 INFO [main-EventThread] regionserver.HRegionServer(2381): Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "HeapMemoryUsage" : { "committed" : 718798848, "init" : 197132288, "max" : 2796552192, "used" : 461039224 }, "NonHeapMemoryUsage" : { "committed" : 104882176, "init" : 2555904, "max" : -1, "used" : 103021264 }, "Verbose" : false, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=IPC", "modelerType" : "RegionServer,sub=IPC", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication", "modelerType" : "RegionServer,sub=Replication", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu" } ], "beans" : [ { "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server", "modelerType" : "RegionServer,sub=Server", "tag.Context" : "regionserver", "tag.Hostname" : "ubuntu" } ] } 2018-01-13 19:32:23,439 INFO [ProcedureDispatcherTimeoutThread] procedure.RSProcedureDispatcher(110): Using procedure batch rpc execution for serverName=ubuntu,38841,1515843136669 version=2097152 2018-01-13 19:32:23,440 DEBUG [RSProcedureDispatcher-pool4-t6] master.ServerManager(783): New admin connection to ubuntu,38841,1515843136669 2018-01-13 19:32:23,442 DEBUG [RSProcedureDispatcher-pool4-t6] ipc.NettyRpcConnection(255): Connecting to ubuntu/127.0.1.1:38841 2018-01-13 19:32:23,448 WARN [RSProcedureDispatcher-pool4-t6] procedure.RSProcedureDispatcher$AbstractRSRemoteCall(204): Failed dispatch to server=ubuntu,38841,1515843136669 try=0 java.net.ConnectException: Call to ubuntu/127.0.1.1:38841 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: ubuntu/127.0.1.1:38841 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:165) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$3(AbstractRpcClient.java:375) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:1) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:92) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1377) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:929) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:179) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$5(NettyRpcConnection.java:176) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:269) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:1) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:659) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:678) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:552) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: ubuntu/127.0.1.1:38841 at org.apache.hbase.thirdparty.io.netty.channel.unix.Socket.finishConnect(..)(Unknown Source) Caused by: org.apache.hbase.thirdparty.io.netty.channel.unix.Errors$NativeConnectException: syscall:getsockopt(..) failed: Connection refused ... 1 more 2018-01-13 19:32:23,448 DEBUG [RS-EventLoopGroup-5-6] ipc.FailedServers(56): Added failed server with address ubuntu/127.0.1.1:38841 to list caused by org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: ubuntu/127.0.1.1:38841 2018-01-13 19:32:23,454 WARN [RSProcedureDispatcher-pool4-t6] assignment.RegionTransitionProcedure(182): Remote call failed pid=8, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1657623790; rit=OPENING, location=ubuntu,38841,1515843136669; exception=Call to ubuntu/127.0.1.1:38841 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: ubuntu/127.0.1.1:38841 2018-01-13 19:32:23,455 INFO [RSProcedureDispatcher-pool4-t6] assignment.AssignProcedure(330): Retry=3 of max=10; pid=8, state=RUNNABLE:REGION_TRANSITION_DISPATCH; AssignProcedure table=hbase:meta, region=1657623790; rit=OPENING, location=ubuntu,38841,1515843136669 2018-01-13 19:32:23,456 INFO [ProcExecWrkr-7] assignment.AssignProcedure(211): Start pid=8, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure table=hbase:meta, region=1657623790; rit=OFFLINE, location=null; forceNewPlan=true, retain=false 2018-01-13 19:32:23,608 WARN [AssignmentThread] assignment.AssignmentManager(1735): no server available, unable to find a location for 1 unassigned regions. waiting 2018-01-13 19:32:24,258 INFO [ubuntu,36079,1515843135537_splitLogManager__Chore_1] hbase.ScheduledChore(180): Chore: SplitLogManager Timeout Monitor was stopped 2018-01-13 19:32:24,610 WARN [AssignmentThread] assignment.AssignmentManager(1735): no server available, unable to find a location for 1 unassigned regions. waiting 2018-01-13 19:32:25,308 WARN [HBase-Metrics2-1] impl.MetricsConfig(125): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2018-01-13 19:32:25,613 WARN [AssignmentThread] assignment.AssignmentManager(1735): no server available, unable to find a location for 1 unassigned regions. waiting 2018-01-13 19:32:26,616 WARN [AssignmentThread] assignment.AssignmentManager(1735): no server available, unable to find a location for 1 unassigned regions. waiting 2018-01-13 19:32:27,619 WARN [AssignmentThread] assignment.AssignmentManager(1735): no server available, unable to find a location for 1 unassigned regions. waiting 2018-01-13 19:32:28,622 WARN [AssignmentThread] assignment.AssignmentManager(1735): no server available, unable to find a location for 1 unassigned regions. waiting 2018-01-13 19:32:29,625 WARN [AssignmentThread] assignment.AssignmentManager(1735): no server available, unable to find a location for 1 unassigned regions. waiting 2018-01-13 19:32:30,628 WARN [AssignmentThread] assignment.AssignmentManager(1735): no server available, unable to find a location for 1 unassigned regions. waiting 2018-01-13 19:32:31,631 WARN [AssignmentThread] assignment.AssignmentManager(1735): no server available, unable to find a location for 1 unassigned regions. waiting 2018-01-13 19:32:32,634 WARN [AssignmentThread] assignment.AssignmentManager(1735): no server available, unable to find a location for 1 unassigned regions. waiting 2018-01-13 19:32:33,636 WARN [AssignmentThread] assignment.AssignmentManager(1735): no server available, unable to find a location for 1 unassigned regions. waiting 2018-01-13 19:32:34,639 WARN [AssignmentThread] assignment.AssignmentManager(1735): no server available, unable to find a location for 1 unassigned regions. waiting 2018-01-13 19:32:35,643 WARN [AssignmentThread] assignment.AssignmentManager(1735): no server available, unable to find a location for 1 unassigned regions. waiting 2018-01-13 19:32:36,646 WARN [AssignmentThread] assignment.AssignmentManager(1735): no server available, unable to find a location for 1 unassigned regions. waiting 2018-01-13 19:32:37,649 WARN [AssignmentThread] assignment.AssignmentManager(1735): no server available, unable to find a location for 1 unassigned regions. waiting 2018-01-13 19:32:38,652 WARN [AssignmentThread] assignment.AssignmentManager(1735): no server available, unable to find a location for 1 unassigned regions. waiting 2018-01-13 19:32:39,654 WARN [AssignmentThread] assignment.AssignmentManager(1735): no server available, unable to find a location for 1 unassigned regions. waiting 2018-01-13 19:32:40,658 WARN [AssignmentThread] assignment.AssignmentManager(1735): no server available, unable to find a location for 1 unassigned regions. waiting 2018-01-13 19:32:41,661 WARN [AssignmentThread] assignment.AssignmentManager(1735): no server available, unable to find a location for 1 unassigned regions. waiting 2018-01-13 19:32:42,663 WARN [AssignmentThread] assignment.AssignmentManager(1735): no server available, unable to find a location for 1 unassigned regions. waiting 2018-01-13 19:32:43,667 WARN [AssignmentThread] assignment.AssignmentManager(1735): no server available, unable to find a location for 1 unassigned regions. waiting 2018-01-13 19:32:44,681 WARN [AssignmentThread] assignment.AssignmentManager(1735): no server available, unable to find a location for 1 unassigned regions. waiting 2018-01-13 19:32:45,684 WARN [AssignmentThread] assignment.AssignmentManager(1735): no server available, unable to find a location for 1 unassigned regions. waiting 2018-01-13 19:32:46,687 WARN [AssignmentThread] assignment.AssignmentManager(1735): no server available, unable to find a location for 1 unassigned regions. waiting 2018-01-13 19:32:47,590 DEBUG [ProcExecTimeout] procedure2.ProcedureExecutor$CompletedProcedureCleaner(221): Evict completed pid=3, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2018-01-13 19:32:47,591 DEBUG [ProcExecTimeout] procedure2.ProcedureExecutor$CompletedProcedureCleaner(221): Evict completed pid=5, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2018-01-13 19:32:47,591 DEBUG [ProcExecTimeout] procedure2.ProcedureExecutor$CompletedProcedureCleaner(221): Evict completed pid=6, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2018-01-13 19:32:47,591 DEBUG [ProcExecTimeout] procedure2.ProcedureExecutor$CompletedProcedureCleaner(221): Evict completed pid=7, state=SUCCESS; AssignProcedure table=hbase:meta, region=534574363 2018-01-13 19:32:47,693 WARN [AssignmentThread] assignment.AssignmentManager(1735): no server available, unable to find a location for 1 unassigned regions. waiting