Sat Jun 13 20:23:06 UTC 2020 Starting master on 085bbffdc52f core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 515492 max locked memory (kbytes, -l) 16384 max memory size (kbytes, -m) unlimited open files (-n) 1048576 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited 2020-06-13 20:23:06,831 INFO [main] master.HMaster: STARTING service HMaster 2020-06-13 20:23:06,834 INFO [main] util.VersionInfo: HBase 3.0.0-SNAPSHOT 2020-06-13 20:23:06,834 INFO [main] util.VersionInfo: Source code repository git://baker221/home/yongle/src/hbase revision=2382f906794ca6b4cac50da32cde707092324bec 2020-06-13 20:23:06,834 INFO [main] util.VersionInfo: Compiled by yongle on Mon May 25 00:34:18 UTC 2020 2020-06-13 20:23:06,834 INFO [main] util.VersionInfo: From source with checksum (stdin)= 2020-06-13 20:23:07,164 INFO [main] master.HMasterCommandLine: Starting a zookeeper cluster 2020-06-13 20:23:07,180 INFO [main] server.ZooKeeperServer: Server environment:zookeeper.version=3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/10/2020 11:30 GMT 2020-06-13 20:23:07,181 INFO [main] server.ZooKeeperServer: Server environment:host.name=085bbffdc52f 2020-06-13 20:23:07,181 INFO [main] server.ZooKeeperServer: Server environment:java.version=1.8.0_252 2020-06-13 20:23:07,181 INFO [main] server.ZooKeeperServer: Server environment:java.vendor=Oracle Corporation 2020-06-13 20:23:07,181 INFO [main] server.ZooKeeperServer: Server environment:java.home=/usr/local/openjdk-8/jre 2020-06-13 20:23:07,181 INFO [main] server.ZooKeeperServer: objenesis-1.0.jar:/hbase/lib/objenesis-2.6.jar:/hbase/lib/okhttp-2.7.5.jar:/hbase/lib/okio-1.6.0.jar:/hbase/lib/osgi-resource-locator-1.0.1.jar:/hbase/lib/paranamer-2.3.jar:/hbase/lib/protobuf-java-2.5.0.jar:/hbase/lib/re2j-1.1.jar:/hbase/lib/snappy-java-1.0.5.jar:/hbase/lib/spymemcached-2.12.2.jar:/hbase/lib/stax2-api-3.1.4.jar:/hbase/lib/token-provider-1.0.1.jar:/hbase/lib/validation-api-1.1.0.Final.jar:/hbase/lib/woodstox-core-5.0.3.jar:/hbase/lib/zookeeper-3.5.7.jar:/hbase/lib/zookeeper-jute-3.5.7.jar:/hbase/lib/client-facing-thirdparty/audience-annotations-0.5.0.jar:/hbase/lib/client-facing-thirdparty/commons-logging-1.2.jar:/hbase/lib/client-facing-thirdparty/htrace-core4-4.2.0-incubating.jar:/hbase/lib/client-facing-thirdparty/jcl-over-slf4j-1.7.30.jar:/hbase/lib/client-facing-thirdparty/jul-to-slf4j-1.7.30.jar:/hbase/lib/client-facing-thirdparty/log4j-1.2.17.jar:/hbase/lib/client-facing-thirdparty/slf4j-api-1.7.30.jar:/hbase/lib/client-facing-thirdparty/slf4j-log4j12-1.7.30.jar 2020-06-13 20:23:07,181 INFO [main] server.ZooKeeperServer: Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2020-06-13 20:23:07,181 INFO [main] server.ZooKeeperServer: Server environment:java.io.tmpdir=/tmp 2020-06-13 20:23:07,181 INFO [main] server.ZooKeeperServer: Server environment:java.compiler= 2020-06-13 20:23:07,181 INFO [main] server.ZooKeeperServer: Server environment:os.name=Linux 2020-06-13 20:23:07,181 INFO [main] server.ZooKeeperServer: Server environment:os.arch=amd64 2020-06-13 20:23:07,181 INFO [main] server.ZooKeeperServer: Server environment:os.version=4.15.0-101-generic 2020-06-13 20:23:07,181 INFO [main] server.ZooKeeperServer: Server environment:user.name=root 2020-06-13 20:23:07,181 INFO [main] server.ZooKeeperServer: Server environment:user.home=/root 2020-06-13 20:23:07,181 INFO [main] server.ZooKeeperServer: Server environment:user.dir=/ 2020-06-13 20:23:07,181 INFO [main] server.ZooKeeperServer: Server environment:os.memory.free=1873MB 2020-06-13 20:23:07,181 INFO [main] server.ZooKeeperServer: Server environment:os.memory.max=30634MB 2020-06-13 20:23:07,181 INFO [main] server.ZooKeeperServer: Server environment:os.memory.total=1948MB 2020-06-13 20:23:07,185 INFO [main] persistence.FileTxnSnapLog: zookeeper.snapshot.trust.empty : false 2020-06-13 20:23:07,203 INFO [main] server.ZKDatabase: zookeeper.snapshotSizeFactor = 0.33 2020-06-13 20:23:07,205 INFO [main] server.ZooKeeperServer: minSessionTimeout set to 4000 2020-06-13 20:23:07,205 INFO [main] server.ZooKeeperServer: maxSessionTimeout set to 40000 2020-06-13 20:23:07,206 INFO [main] server.ZooKeeperServer: Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /var/lib/zookeeper/zookeeper_0/version-2 snapdir /var/lib/zookeeper/zookeeper_0/version-2 2020-06-13 20:23:07,206 INFO [main] server.ZooKeeperServer: minSessionTimeout set to 4000 2020-06-13 20:23:07,206 INFO [main] server.ZooKeeperServer: maxSessionTimeout set to 40000 2020-06-13 20:23:07,219 INFO [main] server.NIOServerCnxnFactory: Configuring NIO connection handler with 10s sessionless connection timeout, 2 selector thread(s), 24 worker threads, and 64 kB direct buffers. 2020-06-13 20:23:07,225 INFO [main] server.NIOServerCnxnFactory: binding to port localhost/127.0.0.1:2181 2020-06-13 20:23:07,233 INFO [main] persistence.FileSnap: Reading snapshot /var/lib/zookeeper/zookeeper_0/version-2/snapshot.0 2020-06-13 20:23:07,262 INFO [main] persistence.FileTxnSnapLog: Snapshotting: 0x41 to /var/lib/zookeeper/zookeeper_0/version-2/snapshot.41 2020-06-13 20:23:07,359 INFO [main] zookeeper.MiniZooKeeperCluster: Started connectionTimeout=30000, dir=/var/lib/zookeeper/zookeeper_0, clientPort=2181, secureClientPort=-1, dataDir=/var/lib/zookeeper/zookeeper_0/version-2, dataDirSize=106565 dataLogDir=/var/lib/zookeeper/zookeeper_0/version-2, dataLogSize=106565 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2020-06-13 20:23:07,362 INFO [main] client.FourLetterWordMain: connecting to localhost 2181 2020-06-13 20:23:07,376 INFO [NIOWorkerThread-1] command.FourLetterCommands: The list of known four letter word commands is : [{1936881266=srvr, 1937006964=stat, 2003003491=wchc, 1685417328=dump, 1668445044=crst, 1936880500=srst, 1701738089=envi, 1668247142=conf, -720899=telnet close, 2003003507=wchs, 2003003504=wchp, 1684632179=dirs, 1668247155=cons, 1835955314=mntr, 1769173615=isro, 1920298859=ruok, 1735683435=gtmk, 1937010027=stmk}] 2020-06-13 20:23:07,376 INFO [NIOWorkerThread-1] command.FourLetterCommands: The list of enabled four letter word commands is : [[wchs, stat, wchp, dirs, stmk, conf, ruok, mntr, srvr, wchc, envi, srst, isro, dump, gtmk, telnet close, crst, cons]] 2020-06-13 20:23:07,377 INFO [NIOWorkerThread-1] server.NIOServerCnxn: Processing stat command from /127.0.0.1:50740 2020-06-13 20:23:07,387 INFO [NIOWorkerThread-1] command.AbstractFourLetterCommand: Stat command output 2020-06-13 20:23:07,388 INFO [main] zookeeper.MiniZooKeeperCluster: Started MiniZooKeeperCluster and ran 'stat' on client port=2181 2020-06-13 20:23:07,389 INFO [main] master.HMasterCommandLine: Starting up instance of localHBaseCluster; master=1, regionserversCount=1 2020-06-13 20:23:07,814 WARN [main] beanutils.FluentPropertyBeanIntrospector: Error when creating PropertyDescriptor for public final void org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)! Ignoring this property. java.beans.IntrospectionException: bad write method arg count: public final void org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object) at java.beans.PropertyDescriptor.findPropertyType(PropertyDescriptor.java:657) at java.beans.PropertyDescriptor.setWriteMethod(PropertyDescriptor.java:327) at java.beans.PropertyDescriptor.(PropertyDescriptor.java:139) at org.apache.commons.beanutils.FluentPropertyBeanIntrospector.createFluentPropertyDescritor(FluentPropertyBeanIntrospector.java:177) at org.apache.commons.beanutils.FluentPropertyBeanIntrospector.introspect(FluentPropertyBeanIntrospector.java:140) at org.apache.commons.beanutils.PropertyUtilsBean.fetchIntrospectionData(PropertyUtilsBean.java:2234) at org.apache.commons.beanutils.PropertyUtilsBean.getIntrospectionData(PropertyUtilsBean.java:2215) at org.apache.commons.beanutils.PropertyUtilsBean.getPropertyDescriptor(PropertyUtilsBean.java:950) at org.apache.commons.beanutils.PropertyUtilsBean.isWriteable(PropertyUtilsBean.java:1466) at org.apache.commons.configuration2.beanutils.BeanHelper.isPropertyWriteable(BeanHelper.java:521) at org.apache.commons.configuration2.beanutils.BeanHelper.initProperty(BeanHelper.java:357) at org.apache.commons.configuration2.beanutils.BeanHelper.initBeanProperties(BeanHelper.java:273) at org.apache.commons.configuration2.beanutils.BeanHelper.initBean(BeanHelper.java:192) at org.apache.commons.configuration2.beanutils.BeanHelper$BeanCreationContextImpl.initBean(BeanHelper.java:669) at org.apache.commons.configuration2.beanutils.DefaultBeanFactory.initBeanInstance(DefaultBeanFactory.java:162) at org.apache.commons.configuration2.beanutils.DefaultBeanFactory.createBean(DefaultBeanFactory.java:116) at org.apache.commons.configuration2.beanutils.BeanHelper.createBean(BeanHelper.java:459) at org.apache.commons.configuration2.beanutils.BeanHelper.createBean(BeanHelper.java:479) at org.apache.commons.configuration2.beanutils.BeanHelper.createBean(BeanHelper.java:492) at org.apache.commons.configuration2.builder.BasicConfigurationBuilder.createResultInstance(BasicConfigurationBuilder.java:447) at org.apache.commons.configuration2.builder.BasicConfigurationBuilder.createResult(BasicConfigurationBuilder.java:417) at org.apache.commons.configuration2.builder.BasicConfigurationBuilder.getConfiguration(BasicConfigurationBuilder.java:285) at org.apache.hadoop.metrics2.impl.MetricsConfig.loadFirst(MetricsConfig.java:119) at org.apache.hadoop.metrics2.impl.MetricsConfig.create(MetricsConfig.java:98) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.configure(MetricsSystemImpl.java:478) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:188) at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:163) at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:62) at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:58) at org.apache.hadoop.hbase.metrics.BaseSourceImpl$DefaultMetricsSystemInitializer.init(BaseSourceImpl.java:54) at org.apache.hadoop.hbase.metrics.BaseSourceImpl.(BaseSourceImpl.java:116) at org.apache.hadoop.hbase.io.MetricsIOSourceImpl.(MetricsIOSourceImpl.java:46) at org.apache.hadoop.hbase.io.MetricsIOSourceImpl.(MetricsIOSourceImpl.java:38) at org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactoryImpl.createIO(MetricsRegionServerSourceFactoryImpl.java:94) at org.apache.hadoop.hbase.io.MetricsIO.(MetricsIO.java:35) at org.apache.hadoop.hbase.io.hfile.HFile.(HFile.java:176) at org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:574) at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:528) at org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.(HMasterCommandLine.java:308) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.util.JVMClusterUtil.createMasterThread(JVMClusterUtil.java:132) at org.apache.hadoop.hbase.LocalHBaseCluster.addMaster(LocalHBaseCluster.java:239) at org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:181) at org.apache.hadoop.hbase.LocalHBaseCluster.(LocalHBaseCluster.java:128) at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:229) at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3077) 2020-06-13 20:23:07,934 INFO [main] metrics.MetricRegistries: Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2020-06-13 20:23:07,971 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2020-06-13 20:23:08,489 INFO [main] regionserver.RSRpcServices: master/085bbffdc52f:16000 server-side Connection retries=45 2020-06-13 20:23:08,532 INFO [main] ipc.RpcExecutor: Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=3, maxQueueLength=300, handlerCount=30 2020-06-13 20:23:08,534 INFO [main] ipc.RpcExecutor: Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=200, handlerCount=20 2020-06-13 20:23:08,535 INFO [main] ipc.RWQueueRpcExecutor: priority.RWQ.Fifo writeQueues=1 writeHandlers=2 readQueues=1 readHandlers=18 scanQueues=0 scanHandlers=0 2020-06-13 20:23:08,535 INFO [main] ipc.RpcExecutor: Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2020-06-13 20:23:08,535 INFO [main] ipc.RpcExecutor: Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=200, handlerCount=1 2020-06-13 20:23:08,847 INFO [main] ipc.RpcServerFactory: Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2020-06-13 20:23:08,910 INFO [main] Configuration.deprecation: hbase.ipc.server.reservoir.initial.buffer.size is deprecated. Instead, use hbase.server.allocator.buffer.size 2020-06-13 20:23:08,911 INFO [main] Configuration.deprecation: hbase.ipc.server.reservoir.initial.max is deprecated. Instead, use hbase.server.allocator.max.buffer.count 2020-06-13 20:23:09,043 INFO [main] ipc.NettyRpcServer: Bind to /250.16.238.1:16000 2020-06-13 20:23:09,301 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=master:16000 connecting to ZooKeeper ensemble=127.0.0.1:2181 2020-06-13 20:23:09,308 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/10/2020 11:30 GMT 2020-06-13 20:23:09,308 INFO [main] zookeeper.ZooKeeper: Client environment:host.name=085bbffdc52f 2020-06-13 20:23:09,308 INFO [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_252 2020-06-13 20:23:09,308 INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 2020-06-13 20:23:09,308 INFO [main] zookeeper.ZooKeeper: Client environment:java.home=/usr/local/openjdk-8/jre 2020-06-13 20:23:09,308 INFO [main] zookeeper.ZooKeeper: objenesis-1.0.jar:/hbase/lib/objenesis-2.6.jar:/hbase/lib/okhttp-2.7.5.jar:/hbase/lib/okio-1.6.0.jar:/hbase/lib/osgi-resource-locator-1.0.1.jar:/hbase/lib/paranamer-2.3.jar:/hbase/lib/protobuf-java-2.5.0.jar:/hbase/lib/re2j-1.1.jar:/hbase/lib/snappy-java-1.0.5.jar:/hbase/lib/spymemcached-2.12.2.jar:/hbase/lib/stax2-api-3.1.4.jar:/hbase/lib/token-provider-1.0.1.jar:/hbase/lib/validation-api-1.1.0.Final.jar:/hbase/lib/woodstox-core-5.0.3.jar:/hbase/lib/zookeeper-3.5.7.jar:/hbase/lib/zookeeper-jute-3.5.7.jar:/hbase/lib/client-facing-thirdparty/audience-annotations-0.5.0.jar:/hbase/lib/client-facing-thirdparty/commons-logging-1.2.jar:/hbase/lib/client-facing-thirdparty/htrace-core4-4.2.0-incubating.jar:/hbase/lib/client-facing-thirdparty/jcl-over-slf4j-1.7.30.jar:/hbase/lib/client-facing-thirdparty/jul-to-slf4j-1.7.30.jar:/hbase/lib/client-facing-thirdparty/log4j-1.2.17.jar:/hbase/lib/client-facing-thirdparty/slf4j-api-1.7.30.jar:/hbase/lib/client-facing-thirdparty/slf4j-log4j12-1.7.30.jar 2020-06-13 20:23:09,308 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2020-06-13 20:23:09,308 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2020-06-13 20:23:09,308 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler= 2020-06-13 20:23:09,308 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux 2020-06-13 20:23:09,308 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64 2020-06-13 20:23:09,308 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=4.15.0-101-generic 2020-06-13 20:23:09,308 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=root 2020-06-13 20:23:09,309 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/root 2020-06-13 20:23:09,309 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/ 2020-06-13 20:23:09,309 INFO [main] zookeeper.ZooKeeper: Client environment:os.memory.free=1593MB 2020-06-13 20:23:09,309 INFO [main] zookeeper.ZooKeeper: Client environment:os.memory.max=30634MB 2020-06-13 20:23:09,309 INFO [main] zookeeper.ZooKeeper: Client environment:os.memory.total=1948MB 2020-06-13 20:23:09,312 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=10000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@644abb8f 2020-06-13 20:23:09,316 INFO [main] common.X509Util: Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation 2020-06-13 20:23:09,320 INFO [main] zookeeper.ClientCnxnSocket: jute.maxbuffer value is 4194304 Bytes 2020-06-13 20:23:09,328 INFO [main] zookeeper.ClientCnxn: zookeeper.request.timeout value is 0. feature enabled= 2020-06-13 20:23:09,337 INFO [main-SendThread(127.0.0.1:2181)] zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2020-06-13 20:23:09,338 INFO [main-SendThread(127.0.0.1:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /127.0.0.1:50742, server: localhost/127.0.0.1:2181 2020-06-13 20:23:09,341 INFO [SyncThread:0] persistence.FileTxnLog: Creating new log file: log.42 2020-06-13 20:23:09,358 INFO [main-SendThread(127.0.0.1:2181)] zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x10084801b670000, negotiated timeout = 10000 2020-06-13 20:23:09,546 INFO [main] util.log: Logging initialized @3181ms 2020-06-13 20:23:09,648 INFO [main] http.HttpServer: Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter) 2020-06-13 20:23:09,648 INFO [main] http.HttpServer: Added global filter 'clickjackingprevention' (class=org.apache.hadoop.hbase.http.ClickjackingPreventionFilter) 2020-06-13 20:23:09,649 INFO [main] http.HttpServer: Added global filter 'securityheaders' (class=org.apache.hadoop.hbase.http.SecurityHeadersFilter) 2020-06-13 20:23:09,652 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context master 2020-06-13 20:23:09,652 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2020-06-13 20:23:09,652 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2020-06-13 20:23:09,659 INFO [main] http.HttpServer: ASYNC_PROFILER_HOME environment variable and async.profiler.home system property not specified. Disabling /prof endpoint. 2020-06-13 20:23:09,682 INFO [main] http.HttpServer: Jetty bound to port 16010 2020-06-13 20:23:09,684 INFO [main] server.Server: jetty-9.3.28.v20191105, build timestamp: 2019-11-05T18:00:51Z, git hash: d7dd68d6e9d8ff06a0130e46886c2db5d70784c1 2020-06-13 20:23:09,729 INFO [main] http.SecurityHeadersFilter: Added security headers filter 2020-06-13 20:23:09,735 INFO [main] handler.ContextHandler: Started o.e.j.s.ServletContextHandler@782168b7{/logs,file:///hbase/logs/,AVAILABLE} 2020-06-13 20:23:09,735 INFO [main] http.SecurityHeadersFilter: Added security headers filter 2020-06-13 20:23:09,735 INFO [main] handler.ContextHandler: Started o.e.j.s.ServletContextHandler@7435a578{/static,file:///hbase/hbase-webapps/static/,AVAILABLE} 2020-06-13 20:23:09,846 INFO [main] webapp.StandardDescriptorProcessor: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet 2020-06-13 20:23:09,858 INFO [main] http.SecurityHeadersFilter: Added security headers filter 2020-06-13 20:23:09,881 INFO [main] handler.ContextHandler: Started o.e.j.w.WebAppContext@7bbbb6a8{/,file:///hbase/hbase-webapps/master/,AVAILABLE}{file:/hbase/hbase-webapps/master} 2020-06-13 20:23:09,887 INFO [main] server.AbstractConnector: Started ServerConnector@421a4ee1{HTTP/1.1,[http/1.1]}{0.0.0.0:16010} 2020-06-13 20:23:09,887 INFO [main] server.Server: Started @3523ms 2020-06-13 20:23:09,892 INFO [main] master.HMaster: hbase.rootdir=file:/var/lib/hbase, hbase.cluster.distributed=false 2020-06-13 20:23:09,968 INFO [main] regionserver.RSRpcServices: regionserver/085bbffdc52f:16020 server-side Connection retries=45 2020-06-13 20:23:09,968 INFO [main] ipc.RpcExecutor: Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=3, maxQueueLength=300, handlerCount=30 2020-06-13 20:23:09,968 INFO [main] ipc.RpcExecutor: Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=200, handlerCount=20 2020-06-13 20:23:09,968 INFO [main] ipc.RWQueueRpcExecutor: priority.RWQ.Fifo writeQueues=1 writeHandlers=2 readQueues=1 readHandlers=18 scanQueues=0 scanHandlers=0 2020-06-13 20:23:09,968 INFO [main] ipc.RpcExecutor: Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2020-06-13 20:23:09,968 INFO [main] ipc.RpcExecutor: Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=200, handlerCount=1 2020-06-13 20:23:09,973 INFO [main] ipc.RpcServerFactory: Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2020-06-13 20:23:09,977 INFO [main] ipc.NettyRpcServer: Bind to /250.16.238.1:16020 2020-06-13 20:23:09,980 INFO [main] Configuration.deprecation: hbase.offheapcache.minblocksize is deprecated. Instead, use hbase.blockcache.minblocksize 2020-06-13 20:23:09,982 INFO [main] hfile.BlockCacheFactory: Allocating BlockCache size=11.97 GB, blockSize=64 KB 2020-06-13 20:23:09,993 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=regionserver:16020 connecting to ZooKeeper ensemble=127.0.0.1:2181 2020-06-13 20:23:09,993 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=10000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@18eec010 2020-06-13 20:23:09,993 INFO [main] zookeeper.ClientCnxnSocket: jute.maxbuffer value is 4194304 Bytes 2020-06-13 20:23:09,994 INFO [main] zookeeper.ClientCnxn: zookeeper.request.timeout value is 0. feature enabled= 2020-06-13 20:23:09,994 INFO [main-SendThread(127.0.0.1:2181)] zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2020-06-13 20:23:09,995 INFO [main-SendThread(127.0.0.1:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /127.0.0.1:50744, server: localhost/127.0.0.1:2181 2020-06-13 20:23:09,997 INFO [main-SendThread(127.0.0.1:2181)] zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x10084801b670001, negotiated timeout = 10000 2020-06-13 20:23:10,024 INFO [main] http.HttpServer: Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter) 2020-06-13 20:23:10,024 INFO [main] http.HttpServer: Added global filter 'clickjackingprevention' (class=org.apache.hadoop.hbase.http.ClickjackingPreventionFilter) 2020-06-13 20:23:10,024 INFO [main] http.HttpServer: Added global filter 'securityheaders' (class=org.apache.hadoop.hbase.http.SecurityHeadersFilter) 2020-06-13 20:23:10,025 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context regionserver 2020-06-13 20:23:10,025 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2020-06-13 20:23:10,026 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2020-06-13 20:23:10,026 INFO [main] http.HttpServer: ASYNC_PROFILER_HOME environment variable and async.profiler.home system property not specified. Disabling /prof endpoint. 2020-06-13 20:23:10,028 INFO [main] http.HttpServer: Jetty bound to port 16030 2020-06-13 20:23:10,028 INFO [main] server.Server: jetty-9.3.28.v20191105, build timestamp: 2019-11-05T18:00:51Z, git hash: d7dd68d6e9d8ff06a0130e46886c2db5d70784c1 2020-06-13 20:23:10,034 INFO [main] http.SecurityHeadersFilter: Added security headers filter 2020-06-13 20:23:10,034 INFO [main] handler.ContextHandler: Started o.e.j.s.ServletContextHandler@6c03fb16{/logs,file:///hbase/logs/,AVAILABLE} 2020-06-13 20:23:10,034 INFO [main] http.SecurityHeadersFilter: Added security headers filter 2020-06-13 20:23:10,034 INFO [main] handler.ContextHandler: Started o.e.j.s.ServletContextHandler@6de0f580{/static,file:///hbase/hbase-webapps/static/,AVAILABLE} 2020-06-13 20:23:10,056 INFO [main] webapp.StandardDescriptorProcessor: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet 2020-06-13 20:23:10,058 INFO [main] http.SecurityHeadersFilter: Added security headers filter 2020-06-13 20:23:10,062 INFO [main] handler.ContextHandler: Started o.e.j.w.WebAppContext@6401d0a0{/,file:///hbase/hbase-webapps/regionserver/,AVAILABLE}{file:/hbase/hbase-webapps/regionserver} 2020-06-13 20:23:10,063 INFO [main] server.AbstractConnector: Started ServerConnector@4ce14f05{HTTP/1.1,[http/1.1]}{0.0.0.0:16030} 2020-06-13 20:23:10,063 INFO [main] server.Server: Started @3699ms 2020-06-13 20:23:10,065 INFO [master/085bbffdc52f:16000:becomeActiveMaster] master.HMaster: Adding backup master ZNode /hbase/backup-masters/085bbffdc52f,16000,1592079787433 2020-06-13 20:23:10,073 INFO [M:0;085bbffdc52f:16000] regionserver.HRegionServer: ClusterId : c94a9a7b-b651-419a-a987-b190d07c78b7 2020-06-13 20:23:10,113 INFO [master/085bbffdc52f:16000:becomeActiveMaster] master.ActiveMasterManager: Another master is the active master, 1d0fd0a03325,16000,1592079697582; waiting to become the next active master 2020-06-13 20:23:19,134 INFO [SessionTracker] server.ZooKeeperServer: Expiring session 0x100847ebc650004, timeout of 10000ms exceeded 2020-06-13 20:23:19,134 INFO [SessionTracker] server.ZooKeeperServer: Expiring session 0x100847ebc650001, timeout of 10000ms exceeded 2020-06-13 20:23:19,134 INFO [SessionTracker] server.ZooKeeperServer: Expiring session 0x100847ebc650000, timeout of 10000ms exceeded 2020-06-13 20:23:19,159 INFO [master/085bbffdc52f:16000:becomeActiveMaster] master.ActiveMasterManager: Deleting ZNode for /hbase/backup-masters/085bbffdc52f,16000,1592079787433 from backup master directory 2020-06-13 20:23:19,169 INFO [master/085bbffdc52f:16000:becomeActiveMaster] master.ActiveMasterManager: Registered as active master=085bbffdc52f,16000,1592079787433 2020-06-13 20:23:19,174 INFO [master/085bbffdc52f:16000:becomeActiveMaster] regionserver.ChunkCreator: Allocating data MemStoreChunkPool with chunk size 2 MB, max count 5514, initial count 0 2020-06-13 20:23:19,176 INFO [master/085bbffdc52f:16000:becomeActiveMaster] regionserver.ChunkCreator: Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 6126, initial count 0 2020-06-13 20:23:19,274 INFO [master/085bbffdc52f:16000:becomeActiveMaster] coordination.SplitLogManagerCoordination: Found 0 orphan tasks and 0 rescan nodes 2020-06-13 20:23:19,274 INFO [master/085bbffdc52f:16000:becomeActiveMaster] hbase.ChoreService: Chore ScheduledChore name=SplitLogManager Timeout Monitor, period=1000, unit=MILLISECONDS is enabled. 2020-06-13 20:23:19,325 INFO [ReadOnlyZKClient-127.0.0.1:2181@0x77bdf832] zookeeper.ZooKeeper: Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=10000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$68/676676346@26fd5ccd 2020-06-13 20:23:19,325 INFO [ReadOnlyZKClient-127.0.0.1:2181@0x77bdf832] zookeeper.ClientCnxnSocket: jute.maxbuffer value is 4194304 Bytes 2020-06-13 20:23:19,326 INFO [ReadOnlyZKClient-127.0.0.1:2181@0x77bdf832] zookeeper.ClientCnxn: zookeeper.request.timeout value is 0. feature enabled= 2020-06-13 20:23:19,327 INFO [ReadOnlyZKClient-127.0.0.1:2181@0x77bdf832-SendThread(127.0.0.1:2181)] zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2020-06-13 20:23:19,327 INFO [ReadOnlyZKClient-127.0.0.1:2181@0x77bdf832-SendThread(127.0.0.1:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /127.0.0.1:50746, server: localhost/127.0.0.1:2181 2020-06-13 20:23:19,333 INFO [ReadOnlyZKClient-127.0.0.1:2181@0x77bdf832-SendThread(127.0.0.1:2181)] zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x10084801b670002, negotiated timeout = 10000 2020-06-13 20:23:19,386 INFO [master/085bbffdc52f:16000:becomeActiveMaster] store.LocalRegion: Create or load local region for table 'master:store', {NAME => 'proc', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2020-06-13 20:23:19,390 INFO [master/085bbffdc52f:16000:becomeActiveMaster] store.LocalRegionFlusherAndCompactor: Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2020-06-13 20:23:19,460 INFO [master/085bbffdc52f:16000:becomeActiveMaster] wal.WALFactory: Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2020-06-13 20:23:19,478 INFO [master/085bbffdc52f:16000:becomeActiveMaster] store.LocalRegion: Renamed file:/var/lib/hbase/MasterData/WALs/1d0fd0a03325,16000,1592079697582 to file:/var/lib/hbase/MasterData/WALs/1d0fd0a03325,16000,1592079697582-dead as it is dead 2020-06-13 20:23:19,481 INFO [master/085bbffdc52f:16000:becomeActiveMaster] store.LocalRegion: Renamed file:/var/lib/hbase/MasterData/WALs/1d0fd0a03325,16000,1592079697582-dead/1d0fd0a03325%2C16000%2C1592079697582.1592079700844 to file:/var/lib/hbase/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/1d0fd0a03325%2C16000%2C1592079697582.1592079700844 2020-06-13 20:23:19,481 INFO [master/085bbffdc52f:16000:becomeActiveMaster] store.LocalRegion: Delete empty local region wal dir file:/var/lib/hbase/MasterData/WALs/1d0fd0a03325,16000,1592079697582-dead 2020-06-13 20:23:19,510 INFO [master/085bbffdc52f:16000:becomeActiveMaster] wal.AbstractFSWAL: WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=085bbffdc52f%2C16000%2C1592079787433, suffix=, logDir=file:/var/lib/hbase/MasterData/WALs/085bbffdc52f,16000,1592079787433, archiveDir=file:/var/lib/hbase/MasterData/oldWALs 2020-06-13 20:23:19,544 INFO [master/085bbffdc52f:16000:becomeActiveMaster] wal.AbstractFSWAL: New WAL /var/lib/hbase/MasterData/WALs/085bbffdc52f,16000,1592079787433/085bbffdc52f%2C16000%2C1592079787433.1592079799521 2020-06-13 20:23:19,652 WARN [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils: Unable to set storagePolicy=HOT for path=file:/var/lib/hbase/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc. DEBUG log level might have more details. java.lang.UnsupportedOperationException: RawLocalFileSystem doesn't support setStoragePolicy at org.apache.hadoop.fs.FileSystem.setStoragePolicy(FileSystem.java:3102) at org.apache.hadoop.fs.FilterFileSystem.setStoragePolicy(FilterFileSystem.java:651) at org.apache.hadoop.hbase.util.CommonFSUtils.invokeSetStoragePolicy(CommonFSUtils.java:519) at org.apache.hadoop.hbase.util.CommonFSUtils.setStoragePolicy(CommonFSUtils.java:502) at org.apache.hadoop.hbase.util.CommonFSUtils.setStoragePolicy(CommonFSUtils.java:475) at org.apache.hadoop.hbase.regionserver.HRegionFileSystem.setStoragePolicy(HRegionFileSystem.java:194) at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:265) at org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:5978) at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1125) at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:1122) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020-06-13 20:23:19,683 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] hfile.CacheConfig: Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false for family {NAME => 'proc', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} with blockCache=null 2020-06-13 20:23:19,695 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration: size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2020-06-13 20:23:19,718 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore: Store=proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2020-06-13 20:23:19,724 INFO [master/085bbffdc52f:16000:becomeActiveMaster] regionserver.HRegion: Replaying edits from file:/var/lib/hbase/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.wals/1d0fd0a03325%2C16000%2C1592079697582.1592079700844 2020-06-13 20:23:19,791 INFO [master/085bbffdc52f:16000:becomeActiveMaster] regionserver.HRegion: Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=41.67 KB heapSize=50.02 KB 2020-06-13 20:23:19,905 INFO [master/085bbffdc52f:16000:becomeActiveMaster] regionserver.DefaultStoreFlusher: Flushed memstore data size=41.67 KB at sequenceid=98 (bloomFilter=true), to=file:/var/lib/hbase/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/b553f42d2a7d4c569b1873a545284857 2020-06-13 20:23:19,939 INFO [master/085bbffdc52f:16000:becomeActiveMaster] regionserver.StoreFileReader: Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b553f42d2a7d4c569b1873a545284857 2020-06-13 20:23:19,961 INFO [master/085bbffdc52f:16000:becomeActiveMaster] regionserver.StoreFileReader: Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b553f42d2a7d4c569b1873a545284857 2020-06-13 20:23:19,962 INFO [master/085bbffdc52f:16000:becomeActiveMaster] regionserver.HStore: Added file:/var/lib/hbase/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/b553f42d2a7d4c569b1873a545284857, entries=110, sequenceid=98, filesize=46.7 K 2020-06-13 20:23:19,969 INFO [master/085bbffdc52f:16000:becomeActiveMaster] regionserver.HRegion: Finished flush of dataSize ~41.67 KB/42673, heapSize ~50.01 KB/51208, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 177ms, sequenceid=98, compaction requested=false; wal=null 2020-06-13 20:23:19,988 INFO [master/085bbffdc52f:16000:becomeActiveMaster] regionserver.HRegion: Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=99 2020-06-13 20:23:19,989 INFO [master/085bbffdc52f:16000:becomeActiveMaster] store.LocalRegionFlusherAndCompactor: Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2020-06-13 20:23:20,019 INFO [master/085bbffdc52f:16000:becomeActiveMaster] region.RegionProcedureStore: Starting the Region Procedure Store, number threads=16 2020-06-13 20:23:20,019 INFO [master/085bbffdc52f:16000:becomeActiveMaster] procedure2.ProcedureExecutor: Starting 16 core workers (bigger of cpus/4 or 16) with max (burst) worker count=160 2020-06-13 20:23:20,021 INFO [master/085bbffdc52f:16000:becomeActiveMaster] region.RegionProcedureStore: Starting Region Procedure Store lease recovery... 2020-06-13 20:23:20,023 INFO [master/085bbffdc52f:16000:becomeActiveMaster] procedure2.ProcedureExecutor: Recovered RegionProcedureStore lease in 0 msec 2020-06-13 20:23:20,112 INFO [master/085bbffdc52f:16000:becomeActiveMaster] procedure2.ProcedureExecutor: Loaded RegionProcedureStore in 89 msec 2020-06-13 20:23:20,112 INFO [master/085bbffdc52f:16000:becomeActiveMaster] procedure2.RemoteProcedureDispatcher: Instantiated, coreThreads=128 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2020-06-13 20:23:20,155 INFO [master/085bbffdc52f:16000:becomeActiveMaster] master.RegionServerTracker: Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 1 possibly 'live' servers, and 0 'splitting'. 2020-06-13 20:23:20,161 INFO [master/085bbffdc52f:16000:becomeActiveMaster] master.ServerManager: Processing expiration of 1d0fd0a03325,16020,1592079700177 on 085bbffdc52f,16000,1592079787433 2020-06-13 20:23:20,206 INFO [master/085bbffdc52f:16000:becomeActiveMaster] assignment.AssignmentManager: Scheduled SCP pid=12 for 1d0fd0a03325,16020,1592079700177 (carryingMeta=true) 1d0fd0a03325,16020,1592079700177/CRASHED/regionCount=0/lock=java.util.concurrent.locks.ReentrantReadWriteLock@5a4128f8[Write locks = 1, Read locks = 0], oldState=ONLINE. 2020-06-13 20:23:20,260 INFO [master/085bbffdc52f:16000:becomeActiveMaster] master.HMaster: Active/primary master=085bbffdc52f,16000,1592079787433, sessionid=0x10084801b670000, setting cluster-up flag (Was=true) 2020-06-13 20:23:20,287 WARN [master/085bbffdc52f:16000:becomeActiveMaster] snapshot.SnapshotManager: Couldn't delete working snapshot directory: file:/var/lib/hbase/.hbase-snapshot/.tmp 2020-06-13 20:23:20,289 INFO [master/085bbffdc52f:16000:becomeActiveMaster] master.ServerManager: No .lastflushedseqids found atfile:/var/lib/hbase/.lastflushedseqids will record last flushed sequence id for regions by regionserver report all over again 2020-06-13 20:23:20,351 INFO [master/085bbffdc52f:16000:becomeActiveMaster] master.HMaster: hbase:meta {1588230740 state=OPEN, ts=1592079800148, server=1d0fd0a03325,16020,1592079700177} 2020-06-13 20:23:20,382 INFO [RS:0;085bbffdc52f:16020] regionserver.HRegionServer: ClusterId : c94a9a7b-b651-419a-a987-b190d07c78b7 2020-06-13 20:23:20,384 INFO [master/085bbffdc52f:16000:becomeActiveMaster] balancer.BaseLoadBalancer: slop=0.001, systemTablesOnMaster=false 2020-06-13 20:23:20,397 INFO [master/085bbffdc52f:16000:becomeActiveMaster] balancer.StochasticLoadBalancer: Loaded config; maxSteps=1000000, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, CPRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] etc. 2020-06-13 20:23:20,400 INFO [ReadOnlyZKClient-127.0.0.1:2181@0x2b1f31ff] zookeeper.ZooKeeper: Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=10000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$68/676676346@26fd5ccd 2020-06-13 20:23:20,401 INFO [ReadOnlyZKClient-127.0.0.1:2181@0x2b1f31ff] zookeeper.ClientCnxnSocket: jute.maxbuffer value is 4194304 Bytes 2020-06-13 20:23:20,401 INFO [ReadOnlyZKClient-127.0.0.1:2181@0x2b1f31ff] zookeeper.ClientCnxn: zookeeper.request.timeout value is 0. feature enabled= 2020-06-13 20:23:20,402 INFO [ReadOnlyZKClient-127.0.0.1:2181@0x2b1f31ff-SendThread(127.0.0.1:2181)] zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2020-06-13 20:23:20,405 INFO [ReadOnlyZKClient-127.0.0.1:2181@0x2b1f31ff-SendThread(127.0.0.1:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /127.0.0.1:50748, server: localhost/127.0.0.1:2181 2020-06-13 20:23:20,408 INFO [master/085bbffdc52f:16000:becomeActiveMaster] procedure2.TimeoutExecutorThread: ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1592079830408 2020-06-13 20:23:20,412 INFO [master/085bbffdc52f:16000:becomeActiveMaster] cleaner.DirScanPool: Cleaner pool size is 3 2020-06-13 20:23:20,413 INFO [PEWorker-1] procedure.ServerCrashProcedure: Start pid=12, state=RUNNABLE:SERVER_CRASH_START, hasLock=true; ServerCrashProcedure server=1d0fd0a03325,16020,1592079700177, splitWal=true, meta=true 2020-06-13 20:23:20,415 INFO [PEWorker-1] master.SplitLogManager: dead splitlog workers [1d0fd0a03325,16020,1592079700177] 2020-06-13 20:23:20,417 INFO [PEWorker-1] master.SplitLogManager: Started splitting 1 logs in [file:/var/lib/hbase/WALs/1d0fd0a03325,16020,1592079700177-splitting] for [1d0fd0a03325,16020,1592079700177] 2020-06-13 20:23:20,424 INFO [ReadOnlyZKClient-127.0.0.1:2181@0x2b1f31ff-SendThread(127.0.0.1:2181)] zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x10084801b670003, negotiated timeout = 10000 2020-06-13 20:23:20,434 INFO [master/085bbffdc52f:16000:becomeActiveMaster] cleaner.CleanerChore: Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2020-06-13 20:23:20,436 INFO [master/085bbffdc52f:16000:becomeActiveMaster] zookeeper.RecoverableZooKeeper: Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=127.0.0.1:2181 2020-06-13 20:23:20,436 INFO [master/085bbffdc52f:16000:becomeActiveMaster] zookeeper.ZooKeeper: Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=10000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@6e178f96 2020-06-13 20:23:20,436 INFO [master/085bbffdc52f:16000:becomeActiveMaster] zookeeper.ClientCnxnSocket: jute.maxbuffer value is 4194304 Bytes 2020-06-13 20:23:20,436 INFO [master/085bbffdc52f:16000:becomeActiveMaster] zookeeper.ClientCnxn: zookeeper.request.timeout value is 0. feature enabled= 2020-06-13 20:23:20,437 INFO [master/085bbffdc52f:16000:becomeActiveMaster] cleaner.CleanerChore: Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2020-06-13 20:23:20,438 INFO [master/085bbffdc52f:16000:becomeActiveMaster-SendThread(127.0.0.1:2181)] zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2020-06-13 20:23:20,438 INFO [master/085bbffdc52f:16000:becomeActiveMaster] cleaner.CleanerChore: Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2020-06-13 20:23:20,438 INFO [master/085bbffdc52f:16000:becomeActiveMaster-SendThread(127.0.0.1:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /127.0.0.1:50750, server: localhost/127.0.0.1:2181 2020-06-13 20:23:20,439 INFO [master/085bbffdc52f:16000:becomeActiveMaster] cleaner.CleanerChore: Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2020-06-13 20:23:20,439 INFO [master/085bbffdc52f:16000:becomeActiveMaster] cleaner.LogCleaner: Creating 2 old WALs cleaner threads 2020-06-13 20:23:20,441 INFO [master/085bbffdc52f:16000:becomeActiveMaster-SendThread(127.0.0.1:2181)] zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x10084801b670004, negotiated timeout = 10000 2020-06-13 20:23:20,441 INFO [master/085bbffdc52f:16000:becomeActiveMaster] hbase.ChoreService: Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2020-06-13 20:23:20,444 INFO [master/085bbffdc52f:16000:becomeActiveMaster] cleaner.CleanerChore: Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2020-06-13 20:23:20,445 INFO [master/085bbffdc52f:16000:becomeActiveMaster] cleaner.CleanerChore: Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2020-06-13 20:23:20,448 INFO [master/085bbffdc52f:16000:becomeActiveMaster] cleaner.CleanerChore: Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2020-06-13 20:23:20,449 INFO [master/085bbffdc52f:16000:becomeActiveMaster] cleaner.CleanerChore: Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2020-06-13 20:23:20,452 INFO [master/085bbffdc52f:16000:becomeActiveMaster] hbase.ChoreService: Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2020-06-13 20:23:20,452 INFO [master/085bbffdc52f:16000:becomeActiveMaster] master.HMaster: Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2020-06-13 20:23:20,459 INFO [master/085bbffdc52f:16000:becomeActiveMaster] hbase.ChoreService: Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2020-06-13 20:23:20,459 INFO [master/085bbffdc52f:16000:becomeActiveMaster] hbase.ChoreService: Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2020-06-13 20:23:20,459 INFO [master/085bbffdc52f:16000:becomeActiveMaster] master.HMaster: Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1592079790104, completionTime=-1 2020-06-13 20:23:20,459 INFO [master/085bbffdc52f:16000:becomeActiveMaster] master.ServerManager: Waiting on regionserver count=0; waited=0ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=4500ms, lastChange=0ms 2020-06-13 20:23:20,468 INFO [RS:0;085bbffdc52f:16020] regionserver.RegionServerCoprocessorHost: System coprocessor loading is enabled 2020-06-13 20:23:20,468 INFO [RS:0;085bbffdc52f:16020] regionserver.RegionServerCoprocessorHost: Table coprocessor loading is enabled 2020-06-13 20:23:20,471 INFO [RS:0;085bbffdc52f:16020] regionserver.HRegionServer: reportForDuty to master=085bbffdc52f,16000,1592079787433 with port=16020, startcode=1592079789963 2020-06-13 20:23:20,610 INFO [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000] master.ServerManager: Registering regionserver=085bbffdc52f,16020,1592079789963 2020-06-13 20:23:20,643 INFO [RS:0;085bbffdc52f:16020] wal.WALFactory: Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2020-06-13 20:23:20,645 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [085bbffdc52f,16020,1592079789963] 2020-06-13 20:23:20,660 INFO [master/085bbffdc52f:16000:becomeActiveMaster] master.ServerManager: Waiting on regionserver count=1; waited=201ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=4500ms, lastChange=0ms 2020-06-13 20:23:20,694 INFO [RS:0;085bbffdc52f:16020] regionserver.MetricsRegionServerWrapperImpl: Computing regionserver metrics every 5000 milliseconds 2020-06-13 20:23:20,732 INFO [RS:0;085bbffdc52f:16020] regionserver.MemStoreFlusher: globalMemStoreLimit=12.0 G, globalMemStoreLimitLowMark=11.4 G, Offheap=false 2020-06-13 20:23:20,739 INFO [RS:0;085bbffdc52f:16020] throttle.PressureAwareCompactionThroughputController: Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2020-06-13 20:23:20,739 INFO [RS:0;085bbffdc52f:16020] hbase.ChoreService: Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2020-06-13 20:23:20,740 INFO [RS:0;085bbffdc52f:16020] regionserver.HRegionServer: CompactionChecker runs every PT10S 2020-06-13 20:23:20,751 INFO [RS:0;085bbffdc52f:16020] hbase.HealthCheckChore: ExecutorStatusChore runs every 1mins, 0sec 2020-06-13 20:23:20,753 INFO [RS:0;085bbffdc52f:16020] hbase.ChoreService: Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2020-06-13 20:23:20,756 INFO [RS:0;085bbffdc52f:16020] hbase.ChoreService: Chore ScheduledChore name=CompactionChecker, period=10000, unit=MILLISECONDS is enabled. 2020-06-13 20:23:20,756 INFO [RS:0;085bbffdc52f:16020] hbase.ChoreService: Chore ScheduledChore name=MemstoreFlusherChore, period=10000, unit=MILLISECONDS is enabled. 2020-06-13 20:23:20,756 INFO [RS:0;085bbffdc52f:16020] hbase.ChoreService: Chore ScheduledChore name=ExecutorStatusChore, period=60000, unit=MILLISECONDS is enabled. 2020-06-13 20:23:20,756 INFO [RS:0;085bbffdc52f:16020] hbase.ChoreService: Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2020-06-13 20:23:20,756 INFO [RS:0;085bbffdc52f:16020] hbase.ChoreService: Chore ScheduledChore name=MovedRegionsCleaner for region 085bbffdc52f,16020,1592079789963, period=120000, unit=MILLISECONDS is enabled. 2020-06-13 20:23:20,781 INFO [SplitLogWorker-085bbffdc52f:16020] regionserver.SplitLogWorker: SplitLogWorker 085bbffdc52f,16020,1592079789963 starting 2020-06-13 20:23:20,786 INFO [SplitLogWorker-085bbffdc52f:16020] coordination.ZkSplitLogWorkerCoordination: worker 085bbffdc52f,16020,1592079789963 acquired task /hbase/splitWAL/WALs%2F1d0fd0a03325%2C16020%2C1592079700177-splitting%2F1d0fd0a03325%252C16020%252C1592079700177.meta.1592079701975.meta 2020-06-13 20:23:20,790 INFO [main-EventThread] coordination.SplitLogManagerCoordination: Task /hbase/splitWAL/WALs%2F1d0fd0a03325%2C16020%2C1592079700177-splitting%2F1d0fd0a03325%252C16020%252C1592079700177.meta.1592079701975.meta acquired by 085bbffdc52f,16020,1592079789963 2020-06-13 20:23:20,790 INFO [RS:0;085bbffdc52f:16020] regionserver.HeapMemoryManager: Starting, tuneOn=false 2020-06-13 20:23:20,798 INFO [RS:0;085bbffdc52f:16020] hbase.ChoreService: Chore ScheduledChore name=085bbffdc52f,16020,1592079789963-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2020-06-13 20:23:20,831 INFO [RS_LOG_REPLAY_OPS-regionserver/085bbffdc52f:16020-0] wal.WALSplitter: Splitting WAL=file:/var/lib/hbase/WALs/1d0fd0a03325,16020,1592079700177-splitting/1d0fd0a03325%2C16020%2C1592079700177.meta.1592079701975.meta, size=3.4 K (3435 bytes) 2020-06-13 20:23:20,831 INFO [RS:0;085bbffdc52f:16020] regionserver.Replication: 085bbffdc52f,16020,1592079789963 started 2020-06-13 20:23:20,831 INFO [RS:0;085bbffdc52f:16020] regionserver.HRegionServer: Serving as 085bbffdc52f,16020,1592079789963, RpcServer on 085bbffdc52f/250.16.238.1:16020, sessionid=0x10084801b670001 2020-06-13 20:23:20,835 INFO [RS_LOG_REPLAY_OPS-regionserver/085bbffdc52f:16020-0] wal.WALSplitter: Open WAL=file:/var/lib/hbase/WALs/1d0fd0a03325,16020,1592079700177-splitting/1d0fd0a03325%2C16020%2C1592079700177.meta.1592079701975.meta cost 5 ms 2020-06-13 20:23:20,836 INFO [RS:0;085bbffdc52f:16020] quotas.RegionServerRpcQuotaManager: Quota support disabled 2020-06-13 20:23:20,836 INFO [RS:0;085bbffdc52f:16020] quotas.RegionServerSpaceQuotaManager: Quota support disabled, not starting space quota manager. 2020-06-13 20:23:20,890 INFO [RS_LOG_REPLAY_OPS-regionserver/085bbffdc52f:16020-0-Writer-2] wal.RecoveredEditsOutputSink: Creating recovered edits writer path=file:/var/lib/hbase/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-1d0fd0a03325%2C16020%2C1592079700177.meta.1592079701975.meta.temp 2020-06-13 20:23:20,900 INFO [RS_LOG_REPLAY_OPS-regionserver/085bbffdc52f:16020-0] wal.OutputSink: 3 split writer threads finished 2020-06-13 20:23:20,902 INFO [split-log-closeStream--pool14-t1] wal.RecoveredEditsOutputSink: Closed recovered edits writer path=file:/var/lib/hbase/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-1d0fd0a03325%2C16020%2C1592079700177.meta.1592079701975.meta.temp (wrote 10 edits, skipped 0 edits in 8 ms 2020-06-13 20:23:20,903 INFO [split-log-closeStream--pool14-t1] wal.RecoveredEditsOutputSink: Rename recovered edits file:/var/lib/hbase/data/hbase/meta/1588230740/recovered.edits/0000000000000000004-1d0fd0a03325%2C16020%2C1592079700177.meta.1592079701975.meta.temp to file:/var/lib/hbase/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2020-06-13 20:23:20,904 INFO [RS_LOG_REPLAY_OPS-regionserver/085bbffdc52f:16020-0] wal.WALSplitter: Processed 10 edits across 1 regions cost 68 ms; edits skipped=1; WAL=file:/var/lib/hbase/WALs/1d0fd0a03325,16020,1592079700177-splitting/1d0fd0a03325%2C16020%2C1592079700177.meta.1592079701975.meta, size=3.4 K, length=3435, corrupted=false, progress failed=false 2020-06-13 20:23:20,907 INFO [RS_LOG_REPLAY_OPS-regionserver/085bbffdc52f:16020-0] coordination.ZkSplitLogWorkerCoordination: successfully transitioned task /hbase/splitWAL/WALs%2F1d0fd0a03325%2C16020%2C1592079700177-splitting%2F1d0fd0a03325%252C16020%252C1592079700177.meta.1592079701975.meta to final state DONE 085bbffdc52f,16020,1592079789963 2020-06-13 20:23:20,907 INFO [RS_LOG_REPLAY_OPS-regionserver/085bbffdc52f:16020-0] handler.WALSplitterHandler: Worker 085bbffdc52f,16020,1592079789963 done with task org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$ZkSplitTaskDetails@debddd8 in 111ms. Status = DONE 2020-06-13 20:23:20,908 INFO [main-EventThread] coordination.SplitLogManagerCoordination: Task /hbase/splitWAL/WALs%2F1d0fd0a03325%2C16020%2C1592079700177-splitting%2F1d0fd0a03325%252C16020%252C1592079700177.meta.1592079701975.meta entered state=DONE 085bbffdc52f,16020,1592079789963 2020-06-13 20:23:20,910 INFO [main-EventThread] wal.WALSplitUtil: Archived processed log file:/var/lib/hbase/WALs/1d0fd0a03325,16020,1592079700177-splitting/1d0fd0a03325%2C16020%2C1592079700177.meta.1592079701975.meta to file:/var/lib/hbase/oldWALs/1d0fd0a03325%2C16020%2C1592079700177.meta.1592079701975.meta 2020-06-13 20:23:20,911 INFO [main-EventThread] coordination.SplitLogManagerCoordination: Done splitting /hbase/splitWAL/WALs%2F1d0fd0a03325%2C16020%2C1592079700177-splitting%2F1d0fd0a03325%252C16020%252C1592079700177.meta.1592079701975.meta 2020-06-13 20:23:20,913 WARN [PEWorker-1] master.SplitLogManager: Returning success without actually splitting and deleting all the log files in path file:/var/lib/hbase/WALs/1d0fd0a03325,16020,1592079700177-splitting: [DeprecatedRawLocalFileStatus{path=file:/var/lib/hbase/WALs/1d0fd0a03325,16020,1592079700177-splitting/1d0fd0a03325%2C16020%2C1592079700177.1592079704861; isDirectory=false; length=7518; replication=1; blocksize=33554432; modification_time=1592079714000; access_time=1592079704859; owner=; group=; permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false}] java.io.IOException: Directory /var/lib/hbase/WALs/1d0fd0a03325,16020,1592079700177-splitting is not empty at org.apache.hadoop.fs.RawLocalFileSystem.delete(RawLocalFileSystem.java:447) at org.apache.hadoop.fs.ChecksumFileSystem.delete(ChecksumFileSystem.java:650) at org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:278) at org.apache.hadoop.hbase.master.MasterWalManager.splitLog(MasterWalManager.java:351) at org.apache.hadoop.hbase.master.MasterWalManager.splitMetaLog(MasterWalManager.java:289) at org.apache.hadoop.hbase.master.MasterWalManager.splitMetaLog(MasterWalManager.java:281) at org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.splitMetaLogs(ServerCrashProcedure.java:300) at org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.executeFromState(ServerCrashProcedure.java:157) at org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure.executeFromState(ServerCrashProcedure.java:64) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:194) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:959) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1671) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1418) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:79) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1987) 2020-06-13 20:23:20,913 INFO [PEWorker-1] master.SplitLogManager: Finished splitting (more than or equal to) 3.4 K (3435 bytes) in 1 log files in [file:/var/lib/hbase/WALs/1d0fd0a03325,16020,1592079700177-splitting] in 496ms 2020-06-13 20:23:20,930 INFO [PEWorker-1] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2020-06-13 20:23:20,943 INFO [PEWorker-2] procedure.MasterProcedureScheduler: Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2020-06-13 20:23:20,945 INFO [PEWorker-2] assignment.TransitRegionStateProcedure: Starting pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2020-06-13 20:23:21,111 INFO [PEWorker-12] zookeeper.MetaTableLocator: Setting hbase:meta (replicaId=0) location in ZooKeeper as 085bbffdc52f,16020,1592079789963 2020-06-13 20:23:21,121 INFO [PEWorker-12] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure}] 2020-06-13 20:23:21,322 INFO [RS_CLOSE_META-regionserver/085bbffdc52f:16020-0] handler.AssignRegionHandler: Open hbase:meta,,1.1588230740 2020-06-13 20:23:21,322 INFO [RS_CLOSE_META-regionserver/085bbffdc52f:16020-0] wal.WALFactory: Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2020-06-13 20:23:21,324 INFO [RS_CLOSE_META-regionserver/085bbffdc52f:16020-0] wal.AbstractFSWAL: WAL configuration: blocksize=64 MB, rollsize=32 MB, prefix=085bbffdc52f%2C16020%2C1592079789963.meta, suffix=.meta, logDir=file:/var/lib/hbase/WALs/085bbffdc52f,16020,1592079789963, archiveDir=file:/var/lib/hbase/oldWALs 2020-06-13 20:23:21,338 INFO [RS_CLOSE_META-regionserver/085bbffdc52f:16020-0] wal.AbstractFSWAL: New WAL /var/lib/hbase/WALs/085bbffdc52f,16020,1592079789963/085bbffdc52f%2C16020%2C1592079789963.meta.1592079801329.meta 2020-06-13 20:23:21,353 INFO [RS_CLOSE_META-regionserver/085bbffdc52f:16020-0] regionserver.RegionCoprocessorHost: Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2020-06-13 20:23:21,367 INFO [StoreOpener-1588230740-1] hfile.CacheConfig: Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false for family {NAME => 'info', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} with blockCache=LruBlockCache{blockCount=0, currentSize=8.98 MB, freeSize=11.96 GB, maxSize=11.97 GB, heapSize=8.98 MB, minSize=11.37 GB, minFactor=0.95, multiSize=5.68 GB, multiFactor=0.5, singleSize=2.84 GB, singleFactor=0.25} 2020-06-13 20:23:21,368 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration: size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2020-06-13 20:23:21,369 INFO [StoreOpener-1588230740-1] regionserver.HStore: Store=info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2020-06-13 20:23:21,371 INFO [StoreOpener-1588230740-1] hfile.CacheConfig: Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false for family {NAME => 'rep_barrier', VERSIONS => '2147483647', EVICT_BLOCKS_ON_CLOSE => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} with blockCache=LruBlockCache{blockCount=0, currentSize=8.98 MB, freeSize=11.96 GB, maxSize=11.97 GB, heapSize=8.98 MB, minSize=11.37 GB, minFactor=0.95, multiSize=5.68 GB, multiFactor=0.5, singleSize=2.84 GB, singleFactor=0.25} 2020-06-13 20:23:21,372 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration: size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2020-06-13 20:23:21,372 INFO [StoreOpener-1588230740-1] regionserver.HStore: Store=rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2020-06-13 20:23:21,373 INFO [StoreOpener-1588230740-1] hfile.CacheConfig: Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false for family {NAME => 'table', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} with blockCache=LruBlockCache{blockCount=0, currentSize=8.98 MB, freeSize=11.96 GB, maxSize=11.97 GB, heapSize=8.98 MB, minSize=11.37 GB, minFactor=0.95, multiSize=5.68 GB, multiFactor=0.5, singleSize=2.84 GB, singleFactor=0.25} 2020-06-13 20:23:21,374 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration: size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2020-06-13 20:23:21,374 INFO [StoreOpener-1588230740-1] regionserver.HStore: Store=table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2020-06-13 20:23:21,379 INFO [RS_CLOSE_META-regionserver/085bbffdc52f:16020-0] regionserver.HRegion: Replaying edits from file:/var/lib/hbase/data/hbase/meta/1588230740/recovered.edits/0000000000000000013 2020-06-13 20:23:21,388 INFO [RS_CLOSE_META-regionserver/085bbffdc52f:16020-0] regionserver.HRegion: Flushing 1588230740 3/3 column families, dataSize=2.60 KB heapSize=5.14 KB 2020-06-13 20:23:21,411 INFO [RS_CLOSE_META-regionserver/085bbffdc52f:16020-0] regionserver.DefaultStoreFlusher: Flushed memstore data size=2.43 KB at sequenceid=13 (bloomFilter=false), to=file:/var/lib/hbase/data/hbase/meta/1588230740/.tmp/info/09ab290859a94dd38dc290ee7721993a 2020-06-13 20:23:21,427 INFO [RS_CLOSE_META-regionserver/085bbffdc52f:16020-0] regionserver.DefaultStoreFlusher: Flushed memstore data size=176 B at sequenceid=13 (bloomFilter=false), to=file:/var/lib/hbase/data/hbase/meta/1588230740/.tmp/table/0891d2ef0a17449aabfd0dd665891dea 2020-06-13 20:23:21,432 INFO [RS_CLOSE_META-regionserver/085bbffdc52f:16020-0] regionserver.HStore: Added file:/var/lib/hbase/data/hbase/meta/1588230740/info/09ab290859a94dd38dc290ee7721993a, entries=20, sequenceid=13, filesize=7.1 K 2020-06-13 20:23:21,435 INFO [RS_CLOSE_META-regionserver/085bbffdc52f:16020-0] regionserver.HStore: Added file:/var/lib/hbase/data/hbase/meta/1588230740/table/0891d2ef0a17449aabfd0dd665891dea, entries=4, sequenceid=13, filesize=4.7 K 2020-06-13 20:23:21,436 INFO [RS_CLOSE_META-regionserver/085bbffdc52f:16020-0] regionserver.HRegion: Finished flush of dataSize ~2.60 KB/2666, heapSize ~4.86 KB/4976, currentSize=0 B/0 for 1588230740 in 48ms, sequenceid=13, compaction requested=false; wal=null 2020-06-13 20:23:21,451 INFO [RS_CLOSE_META-regionserver/085bbffdc52f:16020-0] regionserver.HRegion: Opened 1588230740; next sequenceid=14 2020-06-13 20:23:21,463 INFO [RS_CLOSE_META-regionserver/085bbffdc52f:16020-0] regionserver.HRegionServer: Post open deploy tasks for hbase:meta,,1.1588230740, openProcId=14, masterSystemTime=1592079801297 2020-06-13 20:23:21,485 INFO [PEWorker-4] zookeeper.MetaTableLocator: Setting hbase:meta (replicaId=0) location in ZooKeeper as 085bbffdc52f,16020,1592079789963 2020-06-13 20:23:21,486 INFO [RS_CLOSE_META-regionserver/085bbffdc52f:16020-0] handler.AssignRegionHandler: Opened hbase:meta,,1.1588230740 2020-06-13 20:23:21,490 INFO [PEWorker-4] procedure2.ProcedureExecutor: Finished subprocedure(s) of pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_OPENED, hasLock=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; resume parent processing. 2020-06-13 20:23:21,490 INFO [PEWorker-4] procedure2.ProcedureExecutor: Finished pid=14, ppid=13, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure in 367 msec 2020-06-13 20:23:21,493 INFO [PEWorker-5] procedure2.ProcedureExecutor: Finished subprocedure(s) of pid=12, state=RUNNABLE:SERVER_CRASH_GET_REGIONS, hasLock=true; ServerCrashProcedure server=1d0fd0a03325,16020,1592079700177, splitWal=true, meta=true; resume parent processing. 2020-06-13 20:23:21,493 INFO [PEWorker-5] procedure2.ProcedureExecutor: Finished pid=13, ppid=12, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 561 msec 2020-06-13 20:23:22,163 INFO [master/085bbffdc52f:16000:becomeActiveMaster] master.ServerManager: Waiting on regionserver count=1; waited=1704ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=4500ms, lastChange=-1503ms 2020-06-13 20:23:23,665 INFO [master/085bbffdc52f:16000:becomeActiveMaster] master.ServerManager: Waiting on regionserver count=1; waited=3206ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=4500ms, lastChange=-3005ms 2020-06-13 20:23:23,852 INFO [RS:0;085bbffdc52f:16020] wal.AbstractFSWAL: WAL configuration: blocksize=64 MB, rollsize=32 MB, prefix=085bbffdc52f%2C16020%2C1592079789963, suffix=, logDir=file:/var/lib/hbase/WALs/085bbffdc52f,16020,1592079789963, archiveDir=file:/var/lib/hbase/oldWALs 2020-06-13 20:23:23,861 INFO [RS:0;085bbffdc52f:16020] wal.AbstractFSWAL: New WAL /var/lib/hbase/WALs/085bbffdc52f,16020,1592079789963/085bbffdc52f%2C16020%2C1592079789963.1592079803855 2020-06-13 20:23:24,968 INFO [master/085bbffdc52f:16000:becomeActiveMaster] master.ServerManager: Finished waiting on RegionServer count=1; waited=4509ms, expected min=1 server(s), max=NO_LIMIT server(s), master is running 2020-06-13 20:23:25,094 INFO [master/085bbffdc52f:16000:becomeActiveMaster] assignment.RegionStateStore: Load hbase:meta entry region=3ae6186d5ec8cf43ba174ba776026c64, regionState=OPEN, lastHost=1d0fd0a03325,16020,1592079700177, regionLocation=1d0fd0a03325,16020,1592079700177, openSeqNum=2 2020-06-13 20:23:25,095 INFO [master/085bbffdc52f:16000:becomeActiveMaster] assignment.RegionStateStore: Load hbase:meta entry region=5aafd93485e66431a343bd359fe4fed5, regionState=OPEN, lastHost=1d0fd0a03325,16020,1592079700177, regionLocation=1d0fd0a03325,16020,1592079700177, openSeqNum=2 2020-06-13 20:23:25,095 INFO [master/085bbffdc52f:16000:becomeActiveMaster] assignment.AssignmentManager: Number of RegionServers=1 2020-06-13 20:23:25,095 INFO [master/085bbffdc52f:16000:becomeActiveMaster] procedure2.TimeoutExecutorThread: ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1592079865095 2020-06-13 20:23:25,095 INFO [master/085bbffdc52f:16000:becomeActiveMaster] procedure2.TimeoutExecutorThread: ADDED pid=-1, state=WAITING_TIMEOUT, hasLock=false; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1592079925095 2020-06-13 20:23:25,096 INFO [master/085bbffdc52f:16000:becomeActiveMaster] assignment.AssignmentManager: Joined the cluster in 127 msec 2020-06-13 20:23:25,096 INFO [PEWorker-7] procedure.ServerCrashProcedure: 1d0fd0a03325,16020,1592079700177 had 2 regions 2020-06-13 20:23:25,099 INFO [PEWorker-7] master.SplitLogManager: dead splitlog workers [1d0fd0a03325,16020,1592079700177] 2020-06-13 20:23:25,101 INFO [PEWorker-7] master.SplitLogManager: Started splitting 1 logs in [file:/var/lib/hbase/WALs/1d0fd0a03325,16020,1592079700177-splitting] for [1d0fd0a03325,16020,1592079700177] 2020-06-13 20:23:25,106 INFO [SplitLogWorker-085bbffdc52f:16020] coordination.ZkSplitLogWorkerCoordination: worker 085bbffdc52f,16020,1592079789963 acquired task /hbase/splitWAL/WALs%2F1d0fd0a03325%2C16020%2C1592079700177-splitting%2F1d0fd0a03325%252C16020%252C1592079700177.1592079704861 2020-06-13 20:23:25,107 INFO [main-EventThread] coordination.SplitLogManagerCoordination: Task /hbase/splitWAL/WALs%2F1d0fd0a03325%2C16020%2C1592079700177-splitting%2F1d0fd0a03325%252C16020%252C1592079700177.1592079704861 acquired by 085bbffdc52f,16020,1592079789963 2020-06-13 20:23:25,128 INFO [RS_LOG_REPLAY_OPS-regionserver/085bbffdc52f:16020-1] wal.WALSplitter: Splitting WAL=file:/var/lib/hbase/WALs/1d0fd0a03325,16020,1592079700177-splitting/1d0fd0a03325%2C16020%2C1592079700177.1592079704861, size=7.3 K (7518 bytes) 2020-06-13 20:23:25,141 INFO [RS_LOG_REPLAY_OPS-regionserver/085bbffdc52f:16020-1] wal.WALSplitter: Open WAL=file:/var/lib/hbase/WALs/1d0fd0a03325,16020,1592079700177-splitting/1d0fd0a03325%2C16020%2C1592079700177.1592079704861 cost 13 ms 2020-06-13 20:23:25,160 INFO [RS_LOG_REPLAY_OPS-regionserver/085bbffdc52f:16020-1-Writer-0] wal.RecoveredEditsOutputSink: Creating recovered edits writer path=file:/var/lib/hbase/data/default/TestTable/3ae6186d5ec8cf43ba174ba776026c64/recovered.edits/0000000000000000004-1d0fd0a03325%2C16020%2C1592079700177.1592079704861.temp 2020-06-13 20:23:25,160 INFO [RS_LOG_REPLAY_OPS-regionserver/085bbffdc52f:16020-1-Writer-2] wal.RecoveredEditsOutputSink: Creating recovered edits writer path=file:/var/lib/hbase/data/hbase/namespace/5aafd93485e66431a343bd359fe4fed5/recovered.edits/0000000000000000004-1d0fd0a03325%2C16020%2C1592079700177.1592079704861.temp 2020-06-13 20:23:25,161 INFO [RS_LOG_REPLAY_OPS-regionserver/085bbffdc52f:16020-1] wal.OutputSink: 3 split writer threads finished 2020-06-13 20:23:25,163 INFO [split-log-closeStream--pool15-t1] wal.RecoveredEditsOutputSink: Closed recovered edits writer path=file:/var/lib/hbase/data/default/TestTable/3ae6186d5ec8cf43ba174ba776026c64/recovered.edits/0000000000000000004-1d0fd0a03325%2C16020%2C1592079700177.1592079704861.temp (wrote 1 edits, skipped 0 edits in 0 ms 2020-06-13 20:23:25,164 INFO [split-log-closeStream--pool15-t2] wal.RecoveredEditsOutputSink: Closed recovered edits writer path=file:/var/lib/hbase/data/hbase/namespace/5aafd93485e66431a343bd359fe4fed5/recovered.edits/0000000000000000004-1d0fd0a03325%2C16020%2C1592079700177.1592079704861.temp (wrote 2 edits, skipped 0 edits in 0 ms 2020-06-13 20:23:25,165 INFO [split-log-closeStream--pool15-t1] wal.RecoveredEditsOutputSink: Rename recovered edits file:/var/lib/hbase/data/default/TestTable/3ae6186d5ec8cf43ba174ba776026c64/recovered.edits/0000000000000000004-1d0fd0a03325%2C16020%2C1592079700177.1592079704861.temp to file:/var/lib/hbase/data/default/TestTable/3ae6186d5ec8cf43ba174ba776026c64/recovered.edits/0000000000000000004 2020-06-13 20:23:25,165 INFO [split-log-closeStream--pool15-t2] wal.RecoveredEditsOutputSink: Rename recovered edits file:/var/lib/hbase/data/hbase/namespace/5aafd93485e66431a343bd359fe4fed5/recovered.edits/0000000000000000004-1d0fd0a03325%2C16020%2C1592079700177.1592079704861.temp to file:/var/lib/hbase/data/hbase/namespace/5aafd93485e66431a343bd359fe4fed5/recovered.edits/0000000000000000005 2020-06-13 20:23:25,165 INFO [RS_LOG_REPLAY_OPS-regionserver/085bbffdc52f:16020-1] wal.WALSplitter: Processed 3 edits across 2 regions cost 23 ms; edits skipped=2; WAL=file:/var/lib/hbase/WALs/1d0fd0a03325,16020,1592079700177-splitting/1d0fd0a03325%2C16020%2C1592079700177.1592079704861, size=7.3 K, length=7518, corrupted=false, progress failed=false 2020-06-13 20:23:25,168 INFO [RS_LOG_REPLAY_OPS-regionserver/085bbffdc52f:16020-1] coordination.ZkSplitLogWorkerCoordination: successfully transitioned task /hbase/splitWAL/WALs%2F1d0fd0a03325%2C16020%2C1592079700177-splitting%2F1d0fd0a03325%252C16020%252C1592079700177.1592079704861 to final state DONE 085bbffdc52f,16020,1592079789963 2020-06-13 20:23:25,168 INFO [RS_LOG_REPLAY_OPS-regionserver/085bbffdc52f:16020-1] handler.WALSplitterHandler: Worker 085bbffdc52f,16020,1592079789963 done with task org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$ZkSplitTaskDetails@24a82d95 in 61ms. Status = DONE 2020-06-13 20:23:25,169 INFO [main-EventThread] coordination.SplitLogManagerCoordination: Task /hbase/splitWAL/WALs%2F1d0fd0a03325%2C16020%2C1592079700177-splitting%2F1d0fd0a03325%252C16020%252C1592079700177.1592079704861 entered state=DONE 085bbffdc52f,16020,1592079789963 2020-06-13 20:23:25,171 INFO [main-EventThread] wal.WALSplitUtil: Archived processed log file:/var/lib/hbase/WALs/1d0fd0a03325,16020,1592079700177-splitting/1d0fd0a03325%2C16020%2C1592079700177.1592079704861 to file:/var/lib/hbase/oldWALs/1d0fd0a03325%2C16020%2C1592079700177.1592079704861 2020-06-13 20:23:25,172 INFO [main-EventThread] coordination.SplitLogManagerCoordination: Done splitting /hbase/splitWAL/WALs%2F1d0fd0a03325%2C16020%2C1592079700177-splitting%2F1d0fd0a03325%252C16020%252C1592079700177.1592079704861 2020-06-13 20:23:25,172 INFO [PEWorker-7] master.SplitLogManager: Finished splitting (more than or equal to) 7.3 K (7518 bytes) in 1 log files in [file:/var/lib/hbase/WALs/1d0fd0a03325,16020,1592079700177-splitting] in 71ms 2020-06-13 20:23:25,175 INFO [PEWorker-7] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=TestTable, region=3ae6186d5ec8cf43ba174ba776026c64, ASSIGN}, {pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=hbase:namespace, region=5aafd93485e66431a343bd359fe4fed5, ASSIGN}] 2020-06-13 20:23:25,177 INFO [PEWorker-8] procedure.MasterProcedureScheduler: Took xlock for pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=hbase:namespace, region=5aafd93485e66431a343bd359fe4fed5, ASSIGN 2020-06-13 20:23:25,178 INFO [PEWorker-6] procedure.MasterProcedureScheduler: Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=false; TransitRegionStateProcedure table=TestTable, region=3ae6186d5ec8cf43ba174ba776026c64, ASSIGN 2020-06-13 20:23:25,180 INFO [PEWorker-8] assignment.TransitRegionStateProcedure: Starting pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=true; TransitRegionStateProcedure table=hbase:namespace, region=5aafd93485e66431a343bd359fe4fed5, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2020-06-13 20:23:25,181 INFO [PEWorker-6] assignment.TransitRegionStateProcedure: Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, hasLock=true; TransitRegionStateProcedure table=TestTable, region=3ae6186d5ec8cf43ba174ba776026c64, ASSIGN; state=OPEN, location=null; forceNewPlan=true, retain=false 2020-06-13 20:23:25,185 INFO [master/085bbffdc52f:16000:becomeActiveMaster] hbase.ChoreService: Chore ScheduledChore name=085bbffdc52f,16000,1592079787433-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2020-06-13 20:23:25,186 INFO [master/085bbffdc52f:16000:becomeActiveMaster] hbase.ChoreService: Chore ScheduledChore name=085bbffdc52f,16000,1592079787433-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2020-06-13 20:23:25,186 INFO [master/085bbffdc52f:16000:becomeActiveMaster] hbase.ChoreService: Chore ScheduledChore name=085bbffdc52f,16000,1592079787433-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2020-06-13 20:23:25,189 INFO [master/085bbffdc52f:16000:becomeActiveMaster] hbase.ChoreService: Chore ScheduledChore name=CatalogJanitor-085bbffdc52f:16000, period=300000, unit=MILLISECONDS is enabled. 2020-06-13 20:23:25,189 INFO [master/085bbffdc52f:16000:becomeActiveMaster] hbase.ChoreService: Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2020-06-13 20:23:25,189 INFO [master/085bbffdc52f:16000:becomeActiveMaster] hbase.ChoreService: Chore ScheduledChore name=FlushedSequenceIdFlusher, period=10800000, unit=MILLISECONDS is enabled. 2020-06-13 20:23:25,200 WARN [master/085bbffdc52f:16000:becomeActiveMaster] master.HMaster: hbase:namespace,,1592079706927.5aafd93485e66431a343bd359fe4fed5. is NOT online; state={5aafd93485e66431a343bd359fe4fed5 state=OPEN, ts=1592079805181, server=null}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2020-06-13 20:23:25,202 WARN [master/085bbffdc52f:16000.Chore.1] master.CatalogJanitor: unknown_server=1d0fd0a03325,16020,1592079700177/TestTable,,1592079712988.3ae6186d5ec8cf43ba174ba776026c64., unknown_server=1d0fd0a03325,16020,1592079700177/hbase:namespace,,1592079706927.5aafd93485e66431a343bd359fe4fed5. 2020-06-13 20:23:25,202 WARN [master/085bbffdc52f:16000.Chore.1] master.CatalogJanitor: Playing-it-safe skipping merge/split gc'ing of regions from hbase:meta while regions-in-transition (RIT) 2020-06-13 20:23:25,338 INFO [PEWorker-9] assignment.RegionStateStore: pid=15 updating hbase:meta row=3ae6186d5ec8cf43ba174ba776026c64, regionState=OPENING, regionLocation=085bbffdc52f,16020,1592079789963 2020-06-13 20:23:25,338 INFO [PEWorker-10] assignment.RegionStateStore: pid=16 updating hbase:meta row=5aafd93485e66431a343bd359fe4fed5, regionState=OPENING, regionLocation=085bbffdc52f,16020,1592079789963 2020-06-13 20:23:25,378 INFO [PEWorker-10] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure}] 2020-06-13 20:23:25,378 INFO [PEWorker-9] procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=18, ppid=15, state=RUNNABLE, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure}] 2020-06-13 20:23:25,540 INFO [RS_OPEN_REGION-regionserver/085bbffdc52f:16020-0] handler.AssignRegionHandler: Open TestTable,,1592079712988.3ae6186d5ec8cf43ba174ba776026c64. 2020-06-13 20:23:25,541 INFO [RS_OPEN_PRIORITY_REGION-regionserver/085bbffdc52f:16020-0] handler.AssignRegionHandler: Open hbase:namespace,,1592079706927.5aafd93485e66431a343bd359fe4fed5. 2020-06-13 20:23:25,545 INFO [StoreOpener-3ae6186d5ec8cf43ba174ba776026c64-1] hfile.CacheConfig: Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false for family {NAME => 'info0', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536', METADATA => {'IN_MEMORY_COMPACTION' => 'NONE'}} with blockCache=LruBlockCache{blockCount=2, currentSize=8.98 MB, freeSize=11.96 GB, maxSize=11.97 GB, heapSize=8.98 MB, minSize=11.37 GB, minFactor=0.95, multiSize=5.68 GB, multiFactor=0.5, singleSize=2.84 GB, singleFactor=0.25} 2020-06-13 20:23:25,546 INFO [StoreOpener-5aafd93485e66431a343bd359fe4fed5-1] hfile.CacheConfig: Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false for family {NAME => 'info', VERSIONS => '10', EVICT_BLOCKS_ON_CLOSE => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} with blockCache=LruBlockCache{blockCount=2, currentSize=8.98 MB, freeSize=11.96 GB, maxSize=11.97 GB, heapSize=8.98 MB, minSize=11.37 GB, minFactor=0.95, multiSize=5.68 GB, multiFactor=0.5, singleSize=2.84 GB, singleFactor=0.25} 2020-06-13 20:23:25,546 INFO [StoreOpener-3ae6186d5ec8cf43ba174ba776026c64-1] compactions.CompactionConfiguration: size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2020-06-13 20:23:25,547 INFO [StoreOpener-5aafd93485e66431a343bd359fe4fed5-1] compactions.CompactionConfiguration: size [128 MB, 8.00 EB, 8.00 EB); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2020-06-13 20:23:25,547 INFO [StoreOpener-3ae6186d5ec8cf43ba174ba776026c64-1] regionserver.HStore: Store=info0, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2020-06-13 20:23:25,548 INFO [StoreOpener-5aafd93485e66431a343bd359fe4fed5-1] regionserver.HStore: Store=info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2020-06-13 20:23:25,551 INFO [RS_OPEN_REGION-regionserver/085bbffdc52f:16020-0] regionserver.HRegion: Replaying edits from file:/var/lib/hbase/data/default/TestTable/3ae6186d5ec8cf43ba174ba776026c64/recovered.edits/0000000000000000004 2020-06-13 20:23:25,551 INFO [RS_OPEN_PRIORITY_REGION-regionserver/085bbffdc52f:16020-0] regionserver.HRegion: Replaying edits from file:/var/lib/hbase/data/hbase/namespace/5aafd93485e66431a343bd359fe4fed5/recovered.edits/0000000000000000005 2020-06-13 20:23:25,552 INFO [RS_OPEN_PRIORITY_REGION-regionserver/085bbffdc52f:16020-0] regionserver.HRegion: Flushing 5aafd93485e66431a343bd359fe4fed5 1/1 column families, dataSize=78 B heapSize=488 B 2020-06-13 20:23:25,557 INFO [RS_OPEN_REGION-regionserver/085bbffdc52f:16020-0] regionserver.HRegion: Flushing 3ae6186d5ec8cf43ba174ba776026c64 1/1 column families, dataSize=6.05 KB heapSize=13.53 KB 2020-06-13 20:23:25,571 INFO [RS_OPEN_PRIORITY_REGION-regionserver/085bbffdc52f:16020-0] regionserver.DefaultStoreFlusher: Flushed memstore data size=78 B at sequenceid=5 (bloomFilter=true), to=file:/var/lib/hbase/data/hbase/namespace/5aafd93485e66431a343bd359fe4fed5/.tmp/info/8d672ca4c40e4cecac16a76029fe0a34 2020-06-13 20:23:25,577 INFO [RS_OPEN_PRIORITY_REGION-regionserver/085bbffdc52f:16020-0] regionserver.HStore: Added file:/var/lib/hbase/data/hbase/namespace/5aafd93485e66431a343bd359fe4fed5/info/8d672ca4c40e4cecac16a76029fe0a34, entries=2, sequenceid=5, filesize=4.8 K 2020-06-13 20:23:25,578 INFO [RS_OPEN_PRIORITY_REGION-regionserver/085bbffdc52f:16020-0] regionserver.HRegion: Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 5aafd93485e66431a343bd359fe4fed5 in 26ms, sequenceid=5, compaction requested=false; wal=null 2020-06-13 20:23:25,581 INFO [RS_OPEN_REGION-regionserver/085bbffdc52f:16020-0] regionserver.DefaultStoreFlusher: Flushed memstore data size=6.05 KB at sequenceid=4 (bloomFilter=true), to=file:/var/lib/hbase/data/default/TestTable/3ae6186d5ec8cf43ba174ba776026c64/.tmp/info0/b96abfe2265841d28128ca1033f2af8b 2020-06-13 20:23:25,587 INFO [RS_OPEN_PRIORITY_REGION-regionserver/085bbffdc52f:16020-0] regionserver.HRegion: Opened 5aafd93485e66431a343bd359fe4fed5; next sequenceid=6 2020-06-13 20:23:25,588 INFO [RS_OPEN_REGION-regionserver/085bbffdc52f:16020-0] regionserver.HStore: Added file:/var/lib/hbase/data/default/TestTable/3ae6186d5ec8cf43ba174ba776026c64/info0/b96abfe2265841d28128ca1033f2af8b, entries=100, sequenceid=4, filesize=11.1 K 2020-06-13 20:23:25,588 INFO [RS_OPEN_REGION-regionserver/085bbffdc52f:16020-0] regionserver.HRegion: Finished flush of dataSize ~6.05 KB/6200, heapSize ~13.52 KB/13840, currentSize=0 B/0 for 3ae6186d5ec8cf43ba174ba776026c64 in 31ms, sequenceid=4, compaction requested=false; wal=null 2020-06-13 20:23:25,589 INFO [RS_OPEN_PRIORITY_REGION-regionserver/085bbffdc52f:16020-0] regionserver.HRegionServer: Post open deploy tasks for hbase:namespace,,1592079706927.5aafd93485e66431a343bd359fe4fed5., openProcId=17, masterSystemTime=1592079805533 2020-06-13 20:23:25,594 INFO [RS_OPEN_PRIORITY_REGION-regionserver/085bbffdc52f:16020-0] handler.AssignRegionHandler: Opened hbase:namespace,,1592079706927.5aafd93485e66431a343bd359fe4fed5. 2020-06-13 20:23:25,595 INFO [PEWorker-15] assignment.RegionStateStore: pid=16 updating hbase:meta row=5aafd93485e66431a343bd359fe4fed5, regionState=OPEN, openSeqNum=6, regionLocation=085bbffdc52f,16020,1592079789963 2020-06-13 20:23:25,599 INFO [RS_OPEN_REGION-regionserver/085bbffdc52f:16020-0] regionserver.HRegion: Opened 3ae6186d5ec8cf43ba174ba776026c64; next sequenceid=5 2020-06-13 20:23:25,600 INFO [RS_OPEN_REGION-regionserver/085bbffdc52f:16020-0] regionserver.HRegionServer: Post open deploy tasks for TestTable,,1592079712988.3ae6186d5ec8cf43ba174ba776026c64., openProcId=18, masterSystemTime=1592079805533 2020-06-13 20:23:25,604 INFO [RS_OPEN_REGION-regionserver/085bbffdc52f:16020-0] handler.AssignRegionHandler: Opened TestTable,,1592079712988.3ae6186d5ec8cf43ba174ba776026c64. 2020-06-13 20:23:25,604 INFO [PEWorker-15] procedure2.ProcedureExecutor: Finished subprocedure(s) of pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_OPENED, hasLock=true; TransitRegionStateProcedure table=hbase:namespace, region=5aafd93485e66431a343bd359fe4fed5, ASSIGN; resume parent processing. 2020-06-13 20:23:25,604 INFO [PEWorker-15] procedure2.ProcedureExecutor: Finished pid=17, ppid=16, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure in 223 msec 2020-06-13 20:23:25,605 INFO [PEWorker-14] assignment.RegionStateStore: pid=15 updating hbase:meta row=3ae6186d5ec8cf43ba174ba776026c64, regionState=OPEN, openSeqNum=5, regionLocation=085bbffdc52f,16020,1592079789963 2020-06-13 20:23:25,607 INFO [PEWorker-1] procedure2.ProcedureExecutor: Finished pid=16, ppid=12, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=hbase:namespace, region=5aafd93485e66431a343bd359fe4fed5, ASSIGN in 429 msec 2020-06-13 20:23:25,615 INFO [PEWorker-14] procedure2.ProcedureExecutor: Finished subprocedure(s) of pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_OPENED, hasLock=true; TransitRegionStateProcedure table=TestTable, region=3ae6186d5ec8cf43ba174ba776026c64, ASSIGN; resume parent processing. 2020-06-13 20:23:25,615 INFO [PEWorker-14] procedure2.ProcedureExecutor: Finished pid=18, ppid=15, state=SUCCESS, hasLock=false; org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure in 233 msec 2020-06-13 20:23:25,617 INFO [PEWorker-2] procedure2.ProcedureExecutor: Finished subprocedure(s) of pid=12, state=RUNNABLE:SERVER_CRASH_FINISH, hasLock=true; ServerCrashProcedure server=1d0fd0a03325,16020,1592079700177, splitWal=true, meta=true; resume parent processing. 2020-06-13 20:23:25,617 INFO [PEWorker-12] procedure.ServerCrashProcedure: removed crashed server 1d0fd0a03325,16020,1592079700177 after splitting done 2020-06-13 20:23:25,617 INFO [PEWorker-2] procedure2.ProcedureExecutor: Finished pid=15, ppid=12, state=SUCCESS, hasLock=false; TransitRegionStateProcedure table=TestTable, region=3ae6186d5ec8cf43ba174ba776026c64, ASSIGN in 440 msec 2020-06-13 20:23:25,623 INFO [PEWorker-12] procedure2.ProcedureExecutor: Finished pid=12, state=SUCCESS, hasLock=false; ServerCrashProcedure server=1d0fd0a03325,16020,1592079700177, splitWal=true, meta=true in 5.4530 sec 2020-06-13 20:23:26,359 WARN [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=16020] regionserver.HRegion: No such column family in batch mutation in region org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation@6d5a3592 org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family ns does not exist in region hbase:meta,,1.1588230740 in table 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', REGION_REPLICATION => '1', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'rep_barrier', VERSIONS => '2147483647', EVICT_BLOCKS_ON_CLOSE => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'table', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} at org.apache.hadoop.hbase.regionserver.HRegion.checkFamily(HRegion.java:4640) at org.apache.hadoop.hbase.regionserver.HRegion.checkFamily(HRegion.java:4627) at org.apache.hadoop.hbase.regionserver.HRegion.checkFamilies(HRegion.java:4621) at org.apache.hadoop.hbase.regionserver.HRegion.access$700(HRegion.java:237) at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.checkAndPreparePut(HRegion.java:3674) at org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.checkAndPrepareMutation(HRegion.java:3341) at org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.checkAndPrepareMutation(HRegion.java:3351) at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation$1.visit(HRegion.java:3700) at org.apache.hadoop.hbase.regionserver.HRegion$BatchOperation.visitBatchOperations(HRegion.java:3234) at org.apache.hadoop.hbase.regionserver.HRegion$MutationBatchOperation.checkAndPrepare(HRegion.java:3680) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4120) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4054) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1073) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:1005) at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:968) at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2880) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44870) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:395) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318) 2020-06-13 20:23:26,360 WARN [RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=16020] regionserver.HRegion: REFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'}, {NAME => 'rep_barrier', VERSIONS => '2147483647', EVICT_BLOCKS_ON_CLOSE => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'table', VERSIONS => '3', EVICT_BLOCKS_ON_CLOSE => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'NONE', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'true', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '8192'} 2020-06-13 20:23:26,409 ERROR [master/085bbffdc52f:16000:becomeActiveMaster] master.HMaster: Failed to become active master java.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILED at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:379) at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:319) at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1362) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1137) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 2 actions: RetriesExhaustedException: 2 times, servers with issues: at org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.makeError(BufferedMutatorOverAsyncBufferedMutator.java:107) at org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.internalFlush(BufferedMutatorOverAsyncBufferedMutator.java:122) at org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.close(BufferedMutatorOverAsyncBufferedMutator.java:166) at org.apache.hadoop.hbase.master.TableNamespaceManager.migrateNamespaceTable(TableNamespaceManager.java:93) at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:123) at org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:61) at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:249) at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1360) ... 4 more 2020-06-13 20:23:26,410 ERROR [master/085bbffdc52f:16000:becomeActiveMaster] master.HMaster: Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2020-06-13 20:23:26,410 ERROR [master/085bbffdc52f:16000:becomeActiveMaster] master.HMaster: ***** ABORTING master 085bbffdc52f,16000,1592079787433: Unhandled exception. Starting shutdown. ***** java.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILED at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:379) at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:319) at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1362) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1137) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 2 actions: RetriesExhaustedException: 2 times, servers with issues: at org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.makeError(BufferedMutatorOverAsyncBufferedMutator.java:107) at org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.internalFlush(BufferedMutatorOverAsyncBufferedMutator.java:122) at org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.close(BufferedMutatorOverAsyncBufferedMutator.java:166) at org.apache.hadoop.hbase.master.TableNamespaceManager.migrateNamespaceTable(TableNamespaceManager.java:93) at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:123) at org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:61) at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:249) at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1360) ... 4 more 2020-06-13 20:23:26,410 INFO [master/085bbffdc52f:16000:becomeActiveMaster] regionserver.HRegionServer: ***** STOPPING region server '085bbffdc52f,16000,1592079787433' ***** 2020-06-13 20:23:26,410 INFO [master/085bbffdc52f:16000:becomeActiveMaster] regionserver.HRegionServer: STOPPED: Stopped by master/085bbffdc52f:16000:becomeActiveMaster 2020-06-13 20:23:26,410 INFO [M:0;085bbffdc52f:16000] regionserver.HRegionServer: Stopping infoServer 2020-06-13 20:23:26,419 INFO [M:0;085bbffdc52f:16000] handler.ContextHandler: Stopped o.e.j.w.WebAppContext@7bbbb6a8{/,null,UNAVAILABLE}{file:/hbase/hbase-webapps/master} 2020-06-13 20:23:26,424 INFO [M:0;085bbffdc52f:16000] server.AbstractConnector: Stopped ServerConnector@421a4ee1{HTTP/1.1,[http/1.1]}{0.0.0.0:16010} 2020-06-13 20:23:26,425 INFO [M:0;085bbffdc52f:16000] handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@7435a578{/static,file:///hbase/hbase-webapps/static/,UNAVAILABLE} 2020-06-13 20:23:26,425 INFO [M:0;085bbffdc52f:16000] handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@782168b7{/logs,file:///hbase/logs/,UNAVAILABLE} 2020-06-13 20:23:26,426 INFO [M:0;085bbffdc52f:16000] regionserver.HRegionServer: aborting server 085bbffdc52f,16000,1592079787433 2020-06-13 20:23:26,427 INFO [M:0;085bbffdc52f:16000] regionserver.HRegionServer: stopping server 085bbffdc52f,16000,1592079787433; all regions closed. 2020-06-13 20:23:26,428 INFO [M:0;085bbffdc52f:16000] hbase.ChoreService: Chore service for: master/085bbffdc52f:16000 had [ScheduledChore name=FlushedSequenceIdFlusher, period=10800000, unit=MILLISECONDS] on shutdown 2020-06-13 20:23:26,434 WARN [M:0;085bbffdc52f:16000] master.ActiveMasterManager: Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2020-06-13 20:23:26,434 INFO [M:0;085bbffdc52f:16000] master.ServerManager: Writing .lastflushedseqids file at: file:/var/lib/hbase/.lastflushedseqids 2020-06-13 20:23:26,450 INFO [M:0;085bbffdc52f:16000] assignment.AssignmentManager: Stopping assignment manager 2020-06-13 20:23:26,451 INFO [M:0;085bbffdc52f:16000] procedure2.RemoteProcedureDispatcher: Stopping procedure remote dispatcher 2020-06-13 20:23:26,451 INFO [M:0;085bbffdc52f:16000] procedure2.ProcedureExecutor: Stopping 2020-06-13 20:23:26,457 INFO [M:0;085bbffdc52f:16000] region.RegionProcedureStore: Stopping the Region Procedure Store, isAbort=true 2020-06-13 20:23:26,457 INFO [M:0;085bbffdc52f:16000] store.LocalRegion: Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=true 2020-06-13 20:23:26,463 INFO [M:0;085bbffdc52f:16000] regionserver.HRegion: Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2020-06-13 20:23:26,466 ERROR [M:0;085bbffdc52f:16000] regionserver.HRegion: Memstore data size is 21076 in region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2020-06-13 20:23:26,467 INFO [M:0;085bbffdc52f:16000] regionserver.HRegion: Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2020-06-13 20:23:26,467 INFO [M:0;085bbffdc52f:16000] hbase.ChoreService: Chore service for: master/085bbffdc52f:16000.splitLogManager. had [ScheduledChore name=SplitLogManager Timeout Monitor, period=1000, unit=MILLISECONDS] on shutdown 2020-06-13 20:23:26,467 INFO [master:store-WAL-Roller] wal.AbstractWALRoller: LogRoller exiting. 2020-06-13 20:23:26,467 INFO [M:0;085bbffdc52f:16000] flush.MasterFlushTableProcedureManager: stop: server shutting down. 2020-06-13 20:23:26,467 INFO [M:0;085bbffdc52f:16000] ipc.NettyRpcServer: Stopping server on /250.16.238.1:16000 2020-06-13 20:23:26,536 INFO [ReadOnlyZKClient-127.0.0.1:2181@0x77bdf832] zookeeper.ZooKeeper: Session: 0x10084801b670002 closed 2020-06-13 20:23:26,536 INFO [ReadOnlyZKClient-127.0.0.1:2181@0x77bdf832-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x10084801b670002 2020-06-13 20:23:26,635 INFO [M:0;085bbffdc52f:16000] zookeeper.ZooKeeper: Session: 0x10084801b670000 closed 2020-06-13 20:23:26,635 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x10084801b670000 2020-06-13 20:23:26,635 INFO [M:0;085bbffdc52f:16000] regionserver.HRegionServer: Exiting; stopping=085bbffdc52f,16000,1592079787433; zookeeper connection closed. 2020-06-13 20:23:26,636 ERROR [M:0;085bbffdc52f:16000] server.NIOServerCnxnFactory: Thread Thread[M:0;085bbffdc52f:16000,5,main] died java.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl [FAILED] to be TERMINATED, but the service has FAILED at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:379) at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitTerminated(AbstractService.java:352) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:646) at org.apache.hadoop.hbase.master.HMasterCommandLine$LocalHMaster.run(HMasterCommandLine.java:313) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 2 actions: RetriesExhaustedException: 2 times, servers with issues: at org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.makeError(BufferedMutatorOverAsyncBufferedMutator.java:107) at org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.internalFlush(BufferedMutatorOverAsyncBufferedMutator.java:122) at org.apache.hadoop.hbase.client.BufferedMutatorOverAsyncBufferedMutator.close(BufferedMutatorOverAsyncBufferedMutator.java:166) at org.apache.hadoop.hbase.master.TableNamespaceManager.migrateNamespaceTable(TableNamespaceManager.java:93) at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:123) at org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:61) at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:249) at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1360) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1137) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2245) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:626) ... 1 more 2020-06-13 20:23:27,924 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter: Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2020-06-13 20:24:20,528 INFO [ReadOnlyZKClient-127.0.0.1:2181@0x2b1f31ff] zookeeper.ZooKeeper: Session: 0x10084801b670003 closed 2020-06-13 20:24:20,528 INFO [ReadOnlyZKClient-127.0.0.1:2181@0x2b1f31ff-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x10084801b670003 2020-06-13 20:26:40,558 ERROR [main] master.HMasterCommandLine: Master exiting java.lang.RuntimeException: Master not initialized after 200000ms at org.apache.hadoop.hbase.util.JVMClusterUtil.waitForEvent(JVMClusterUtil.java:232) at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:200) at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:433) at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:232) at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3077) 2020-06-13 20:26:40,563 INFO [shutdown-hook-0] regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@46baf579 2020-06-13 20:26:40,563 INFO [shutdown-hook-0] regionserver.HRegionServer: ***** STOPPING region server '085bbffdc52f,16020,1592079789963' ***** 2020-06-13 20:26:40,563 INFO [shutdown-hook-0] regionserver.HRegionServer: STOPPED: Shutdown hook 2020-06-13 20:26:40,595 INFO [RS:0;085bbffdc52f:16020] regionserver.SplitLogWorker: Sending interrupt to stop the worker thread 2020-06-13 20:26:40,595 INFO [RS:0;085bbffdc52f:16020] regionserver.HRegionServer: Stopping infoServer 2020-06-13 20:26:40,595 INFO [SplitLogWorker-085bbffdc52f:16020] regionserver.SplitLogWorker: SplitLogWorker interrupted. Exiting. 2020-06-13 20:26:40,596 INFO [SplitLogWorker-085bbffdc52f:16020] regionserver.SplitLogWorker: SplitLogWorker 085bbffdc52f,16020,1592079789963 exiting 2020-06-13 20:26:40,597 INFO [RS:0;085bbffdc52f:16020] handler.ContextHandler: Stopped o.e.j.w.WebAppContext@6401d0a0{/,null,UNAVAILABLE}{file:/hbase/hbase-webapps/regionserver} 2020-06-13 20:26:40,598 INFO [RS:0;085bbffdc52f:16020] server.AbstractConnector: Stopped ServerConnector@4ce14f05{HTTP/1.1,[http/1.1]}{0.0.0.0:16030} 2020-06-13 20:26:40,599 INFO [RS:0;085bbffdc52f:16020] handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@6de0f580{/static,file:///hbase/hbase-webapps/static/,UNAVAILABLE} 2020-06-13 20:26:40,599 INFO [RS:0;085bbffdc52f:16020] handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@6c03fb16{/logs,file:///hbase/logs/,UNAVAILABLE} 2020-06-13 20:26:40,601 INFO [RS:0;085bbffdc52f:16020] regionserver.HeapMemoryManager: Stopping 2020-06-13 20:26:40,602 INFO [RS:0;085bbffdc52f:16020] flush.RegionServerFlushTableProcedureManager: Stopping region server flush procedure manager gracefully. 2020-06-13 20:26:40,602 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher: MemStoreFlusher.0 exiting 2020-06-13 20:26:40,602 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher: MemStoreFlusher.1 exiting 2020-06-13 20:26:40,602 INFO [RS:0;085bbffdc52f:16020] snapshot.RegionServerSnapshotManager: Stopping RegionServerSnapshotManager gracefully. 2020-06-13 20:26:40,603 INFO [RS:0;085bbffdc52f:16020] regionserver.HRegionServer: Received CLOSE for 5aafd93485e66431a343bd359fe4fed5 2020-06-13 20:26:40,604 INFO [RS:0;085bbffdc52f:16020] regionserver.HRegionServer: Received CLOSE for 3ae6186d5ec8cf43ba174ba776026c64 2020-06-13 20:26:40,604 INFO [RS:0;085bbffdc52f:16020] regionserver.HRegionServer: stopping server 085bbffdc52f,16020,1592079789963 2020-06-13 20:26:40,605 INFO [RS:0;085bbffdc52f:16020] regionserver.CompactSplit: Waiting for Split Thread to finish... 2020-06-13 20:26:40,605 INFO [RS:0;085bbffdc52f:16020] regionserver.CompactSplit: Waiting for Large Compaction Thread to finish... 2020-06-13 20:26:40,605 INFO [RS:0;085bbffdc52f:16020] regionserver.CompactSplit: Waiting for Small Compaction Thread to finish... 2020-06-13 20:26:40,605 INFO [RS:0;085bbffdc52f:16020] regionserver.HRegionServer: Received CLOSE for 1588230740 2020-06-13 20:26:40,605 INFO [RS:0;085bbffdc52f:16020] regionserver.HRegionServer: Waiting on 3 regions to close 2020-06-13 20:26:40,607 INFO [RS_CLOSE_REGION-regionserver/085bbffdc52f:16020-0] regionserver.HRegion: Closing region hbase:namespace,,1592079706927.5aafd93485e66431a343bd359fe4fed5. 2020-06-13 20:26:40,607 INFO [RS_CLOSE_META-regionserver/085bbffdc52f:16020-0] regionserver.HRegion: Closing region hbase:meta,,1.1588230740 2020-06-13 20:26:40,607 INFO [RS_CLOSE_META-regionserver/085bbffdc52f:16020-0] regionserver.HRegion: Flushing 1588230740 3/3 column families, dataSize=1.91 KB heapSize=3.85 KB 2020-06-13 20:26:40,607 INFO [RS_CLOSE_REGION-regionserver/085bbffdc52f:16020-1] regionserver.HRegion: Closing region TestTable,,1592079712988.3ae6186d5ec8cf43ba174ba776026c64. 2020-06-13 20:26:40,627 INFO [RS_CLOSE_REGION-regionserver/085bbffdc52f:16020-1] regionserver.HRegion: Closed TestTable,,1592079712988.3ae6186d5ec8cf43ba174ba776026c64. 2020-06-13 20:26:40,627 INFO [RS_CLOSE_REGION-regionserver/085bbffdc52f:16020-0] regionserver.HRegion: Closed hbase:namespace,,1592079706927.5aafd93485e66431a343bd359fe4fed5. 2020-06-13 20:26:40,640 INFO [RS_CLOSE_META-regionserver/085bbffdc52f:16020-0] regionserver.DefaultStoreFlusher: Flushed memstore data size=1.91 KB at sequenceid=20 (bloomFilter=false), to=file:/var/lib/hbase/data/hbase/meta/1588230740/.tmp/info/373ec0bd2341494bb5e49f2faec9280c 2020-06-13 20:26:40,645 INFO [RS_CLOSE_META-regionserver/085bbffdc52f:16020-0] regionserver.HStore: Added file:/var/lib/hbase/data/hbase/meta/1588230740/info/373ec0bd2341494bb5e49f2faec9280c, entries=16, sequenceid=20, filesize=6.6 K 2020-06-13 20:26:40,646 INFO [RS_CLOSE_META-regionserver/085bbffdc52f:16020-0] regionserver.HRegion: Finished flush of dataSize ~1.91 KB/1956, heapSize ~3.34 KB/3416, currentSize=0 B/0 for 1588230740 in 39ms, sequenceid=20, compaction requested=false 2020-06-13 20:26:40,656 INFO [RS_CLOSE_META-regionserver/085bbffdc52f:16020-0] regionserver.HRegion: Closed hbase:meta,,1.1588230740 2020-06-13 20:26:40,756 INFO [regionserver/085bbffdc52f:16020.Chore.1] hbase.ScheduledChore: Chore: MemstoreFlusherChore was stopped 2020-06-13 20:26:40,756 INFO [regionserver/085bbffdc52f:16020.Chore.1] hbase.ScheduledChore: Chore: CompactionChecker was stopped 2020-06-13 20:26:40,759 INFO [regionserver/085bbffdc52f:16020.leaseChecker] regionserver.LeaseManager: Closed leases 2020-06-13 20:26:40,807 INFO [RS:0;085bbffdc52f:16020] regionserver.HRegionServer: stopping server 085bbffdc52f,16020,1592079789963; all regions closed. 2020-06-13 20:26:40,811 INFO [RS:0;085bbffdc52f:16020] wal.AbstractFSWAL: Closed WAL: AsyncFSWAL 085bbffdc52f%2C16020%2C1592079789963.meta:.meta(num 1592079801329) 2020-06-13 20:26:40,813 INFO [RS:0;085bbffdc52f:16020] wal.AbstractFSWAL: Closed WAL: AsyncFSWAL 085bbffdc52f%2C16020%2C1592079789963:(num 1592079803855) 2020-06-13 20:26:40,813 INFO [RS:0;085bbffdc52f:16020] regionserver.LeaseManager: Closed leases 2020-06-13 20:26:40,813 INFO [RS:0;085bbffdc52f:16020] hbase.ChoreService: Chore service for: regionserver/085bbffdc52f:16020 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2020-06-13 20:26:40,814 INFO [regionserver/085bbffdc52f:16020.logRoller] wal.AbstractWALRoller: LogRoller exiting. 2020-06-13 20:26:40,816 INFO [RS:0;085bbffdc52f:16020] ipc.NettyRpcServer: Stopping server on /250.16.238.1:16020 2020-06-13 20:26:40,930 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x10084801b670001 2020-06-13 20:26:40,930 INFO [RS:0;085bbffdc52f:16020] zookeeper.ZooKeeper: Session: 0x10084801b670001 closed 2020-06-13 20:26:40,930 INFO [RS:0;085bbffdc52f:16020] regionserver.HRegionServer: Exiting; stopping=085bbffdc52f,16020,1592079789963; zookeeper connection closed. 2020-06-13 20:26:40,930 INFO [shutdown-hook-0] regionserver.ShutdownHook: Starting fs shutdown hook thread. 2020-06-13 20:26:40,931 INFO [shutdown-hook-0] regionserver.ShutdownHook: Shutdown hook finished.